netgamut tutorial

netgamut lets you browse, download and cache network datasets from Python — and turn path data into the state networks that Infomap’s memory model needs. Three sources are indexed:

Source	What it is
netgamut.snap	the Stanford Large Network Dataset Collection
netgamut.netzschleuder	networks.skewed.de, tagged by domain
netgamut.paths	curated path data: flight itineraries, navigation trails

pip install "netgamut[all]"   # extras: networkx, infomap

Everything below runs offline except the two small downloads, which land in a shared user-level cache — re-running this notebook fetches nothing twice.

Browse before you download

Metadata for every dataset ships inside the package, mapped onto one domain vocabulary across all sources:

import netgamut

netgamut.domains()

{'Social': 212,
 'Informational': 71,
 'Biological': 36,
 'Technological': 30,
 'Transportation': 17,
 'Economic': 39}

for match in netgamut.search(domain="Transportation"):
    print(match)

snap/roadNet-CA [Transportation] (1,965,206 nodes, 2,766,607 edges)
snap/roadNet-PA [Transportation] (1,088,092 nodes, 1,541,898 edges)
snap/roadNet-TX [Transportation] (1,379,917 nodes, 1,921,660 edges)
netzschleuder/chicago_road [Transportation] (12,982 nodes, 39,018 edges)
netzschleuder/contiguous_usa [Transportation] (49 nodes, 107 edges)
netzschleuder/eu_airlines [Transportation] (450 nodes, 3,588 edges)
netzschleuder/euroroad [Transportation] (1,174 nodes, 1,417 edges)
netzschleuder/faa_routes [Transportation] (1,226 nodes, 2,615 edges)
netzschleuder/london_transport [Transportation] (369 nodes, 441 edges)
netzschleuder/openflights [Transportation] (3,214 nodes, 66,771 edges)
netzschleuder/openstreetmap [Transportation] (351 nodes, 866 edges)
netzschleuder/paris_transportation [Transportation] (42,678 nodes, 51,931 edges)
netzschleuder/roadnet [Transportation] (1,971,281 nodes, 5,533,214 edges)
netzschleuder/urban_streets [Transportation] (2,870 nodes, 4,387 edges)
netzschleuder/us_air_traffic [Transportation] (2,278 nodes, 6,390,340 edges)
netzschleuder/us_roads [Transportation] (69,082 nodes, 78,100 edges)
paths/db1b-coupon [Transportation]

Each source also has its own richer search() and a describe() that prints everything the site reports — none of it touches the network itself.

from netgamut import netzschleuder as nz

print(nz.describe("karate"))

karate
======
Domains     Social
Properties  unweighted, undirected
Summary     Network of friendships among members of a university karate club. Includes metadata for faction membership after a social partition. Note: there are two versions of this network, one with 77 edges and one with 78, due to an ambiguous typo in the original study. (The most commonly used is the one with 78 edges.)[^icon]
[^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project
Page        https://networks.skewed.de/karate

Reported measures
  num_vertices                 34
  num_edges                    77
  average_degree               4.529
  global_clustering            0.2583
  degree_assortativity         -0.4675
  diameter                     5
  largest_component_fraction   1

Subgraphs   2 (default 77): 77, 78

Source
  ['W. W. Zachary, "An information flow model for conflict and fission in small groups." Journal of Anthropological Research 33, 452-473 (1977).', 'https://doi.org/10.1086/jar.33.4.3629752']

Load a network and run Infomap

load() downloads on first use, caches, and returns a lazy handle; to_infomap() hands the edges to the Infomap Python API:

net = nz.load("karate", progress=False)
result = net.to_infomap(silent=True, num_trials=5, seed=1).run()
print(f"{result.num_top_modules} modules, codelength {result.codelength:.3f} bits")

3 modules, codelength 4.314 bits

Path data and memory networks

A regular network forgets where flow came from: at node $j$ every walker looks the same. Path data says otherwise. The paths source ships datasets whose unit of observation is a whole trajectory, and converts them into second-order state networks: a sliding window turns every observed trigram $i \to j \to k$ into a link from state node $(i, j)$ to state node $(j, k)$, where $(i, j)$ means at physical node $j$, having arrived from $i$. Infomap reads this representation natively (Rosvall et al., Memory in network flows and its effects on spreading dynamics and community detection, Nat. Commun. 5, 4630, 2014).

Wikispeedia is the small, instant demo: 51,318 human navigation trails between Wikipedia articles, one 9.5 MB download.

from netgamut import paths

data = paths.load("wikispeedia", progress=False)
trails = list(data.paths())
print(f"{len(trails):,} trails, e.g.:")
print("  " + " -> ".join(trails[2][0]))

51,318 trails, e.g.:
  14th_century -> Niger -> Nigeria -> British_Empire -> Slavery -> Africa -> Atlantic_slave_trade -> African_slave_trade

Demo-sized subsets

For demonstrations and small benchmarks, max_paths keeps only the first paths and max_nodes keeps the most-visited physical nodes. Trajectories are cut at removed nodes, never bridged across them, so no transition is fabricated:

memory = data.to_state_network(order=2, max_nodes=300)
print(memory)

StateNetwork(order=2): 300 physical nodes, 6,475 state nodes, 23,411 links from 28,380 paths

First order versus second order

The order parameter generalises the window — order=1 recovers the plain directed network from the same trajectories, so the case for memory is a one-argument comparison on identical data:

for order in (1, 2):
    net_o = data.to_state_network(order=order, max_nodes=300)
    result = net_o.to_infomap(silent=True, num_trials=5, seed=1).run()
    print(
        f"order {order}: {net_o.num_state_nodes:>6,} states, "
        f"{result.num_top_modules:>4} modules, "
        f"codelength {result.codelength:.3f} bits"
    )

order 1:    300 states,   23 modules, codelength 6.594 bits

order 2:  6,475 states,  177 modules, codelength 5.886 bits

With memory, modules can overlap on physical nodes: the same article belongs to different modules depending on where readers came from.

im = memory.to_infomap(silent=True, num_trials=5, seed=1)
im.run()

modules_per_article = {}
for node in im.nodes:  # state-level view: node.node_id is the physical node
    modules_per_article.setdefault(node.node_id, set()).add(node.module_id)

overlapping = sum(1 for modules in modules_per_article.values() if len(modules) > 1)
print(f"{overlapping} of {len(modules_per_article)} articles sit in more than one module")

262 of 300 articles sit in more than one module

Files for the Infomap CLI

write() emits the *Vertices / *States / *Links format, ready for infomap wikispeedia_states.net output/ --directed:

state_file = memory.write("wikispeedia_states.net")
print("\n".join(state_file.read_text().splitlines()[:8]))

# State network with Markov order 2 built from 28380 paths
*Vertices 300
1 "16th_century"
2 "Pacific_Ocean"
3 "Atlantic_Ocean"
4 "British_Empire"
5 "Slavery"
6 "Africa"

Scaling up: flight itineraries

The same three lines work on the BTS DB1B survey — US airline itineraries, the flight data family behind the memory-networks paper. Each itinerary is a sequence of airports weighted by its sampled passengers, and any quarter since 1993 is one keyword away. Not executed here because a quarter is a ~200 MB download (~2 GB unzipped) plus a one-time grouping pass that is cached next to it:

data = paths.load("db1b-coupon", year=2019, quarter=1)
memory = data.to_state_network(order=2, max_nodes=100)  # 100 busiest airports
result = memory.to_infomap(silent=True).run()

Please cite what you use

for name in ("wikispeedia", "db1b-coupon"):
    dataset = paths.info(name)
    print(dataset.name)
    for citation in dataset.citations:
        print("   ", citation)

wikispeedia
    Robert West and Jure Leskovec. Human Wayfinding in Information Networks. WWW, 2012.
    Robert West, Joelle Pineau and Doina Precup. Wikispeedia: An Online Game for Inferring Semantic Distances between Concepts. IJCAI, 2009.
db1b-coupon
    Bureau of Transportation Statistics, US Department of Transportation. Origin and Destination Survey DB1BCoupon. https://transtats.bts.gov

Further reading: the map equation framework and Infomap live at mapequation.org; the memory-networks model is Rosvall et al., Nat. Commun. 5, 4630 (2014), doi:10.1038/ncomms5630.