Sources¶
Each source’s own search() filters on what only it records. The network
and cache classes they re-export are documented once, under
Network objects and Cross-source API, and DataFile and DownloadError
under SNAP.
SNAP¶
Access to the Stanford Large Network Dataset Collection (SNAP).
Metadata is answered from a catalog bundled with the package, so you can browse the whole collection – categories, sizes, statistics, citations – without touching the network:
from netgamut import snap
for category in snap.categories():
print(category)
print(snap.describe("ca-GrQc"))
Only load() downloads anything, and it caches into a shared user-level
directory, so a dataset is fetched once per machine no matter how many
projects, notebooks or virtualenvs use it:
net = snap.load("ca-GrQc")
net.path # a plain text file on disk
sum(1 for _ in net.edges()) # streamed, never fully in memory
SNAP is a third-party collection. Please cite the datasets you use; see
Dataset.citations.
- class netgamut.snap.Catalog(categories, datasets, source='https://snap.stanford.edu/data/index.html', retrieved='', origin='package')[source]¶
An immutable snapshot of the SNAP collection.
- datasets_in(category)[source]¶
Every dataset filed under category.
Cross-listed datasets appear under each of their categories.
- search(query='', *, category=None, directed=None, max_nodes=None, min_nodes=None, max_edges=None, min_edges=None, ground_truth=None)[source]¶
Filter the catalog.
query matches case-insensitively against the dataset name and description. The remaining arguments narrow by category, orientation and size. Datasets that do not report a size are excluded whenever a size bound is given. Set ground_truth to select only datasets that ship a reference partition, which is what you want when assembling a community-detection benchmark.
- class netgamut.snap.Category(id, name, description='', toc_name='', dataset_names=())[source]¶
One of the section headings on the SNAP index page.
- class netgamut.snap.Dataset(name, categories=(), category_names=(), description='', type=None, url='', columns=<factory>, statistics=<factory>, files=(), citations=())[source]¶
Metadata for a single SNAP dataset.
Everything here is available offline, so you can inspect a dataset’s size and shape before deciding to download it.
- categories: tuple[str, ...]¶
Every category SNAP files this dataset under, in page order. Most datasets have one, but the site genuinely cross-lists some: wiki-Vote sits in both Social networks and Wikipedia networks, and soc-RedditHyperlinks in three sections.
- columns: dict[str, str]¶
The dataset’s row in the SNAP index tables, keyed by those tables’ own column headers. Categories use different columns, so this preserves what the page actually said rather than forcing a common schema. For a cross-listed dataset the columns from each section are merged, which is how a temporal network keeps both Static Edges and Temporal Edges.
- property domains: tuple[str, ...]¶
Normalised domains, shared with every other source.
SNAP groups by provenance rather than by domain, so this is a mapping rather than a relabelling – see
netgamut._core.domains. A few SNAP sections mix domains and yield nothing here.
- property edges: int | None¶
Edge count from the index table, or
Noneif not reported.Temporal categories report both static and temporal edges; this returns the static count. Use
columnsfor the full picture.
- property directed: bool | None¶
Truefor directed networks,Falsefor undirected.Nonewhen SNAP does not state a direction, which happens for the item-oriented categories such as Online reviews.
- property weighted: bool | None¶
Whether SNAP’s type for the graph says Weighted;
Noneif it is no graph’s.
- property signed: bool | None¶
Whether SNAP’s type for the graph says Signed;
Noneif it is no graph’s.
- property temporal: bool | None¶
Whether SNAP’s type for the graph says Temporal;
Noneif it is no graph’s.
- property bipartite: bool | None¶
Whether SNAP’s type for the graph says Bipartite;
Noneif it is no graph’s.
- property primary_file: DataFile | None¶
The file
netgamut.snap.load()downloads by default.Prefers a plain edge list named after the dataset, then any gzipped text file, then the first non-readme artifact.
- property community_files: tuple[DataFile, ...]¶
Overlapping ground-truth community lists, if SNAP publishes any.
One line per community, listing its node ids. The
com-*datasets use this shape; communities overlap, so scoring needs an overlap-aware measure.
- property label_files: tuple[DataFile, ...]¶
Node-label ground truth, if SNAP publishes any.
One
node labelpair per line –email-Eu-corelabels each person with their department. This shape is a hard partition, so it can be scored directly against a community-detection run with NMI or AMI.
- property ground_truth_kind: str | None¶
"communities","labels", orNone.The two shapes need different scoring, so check this before choosing a metric.
- community_file(variant='all')[source]¶
Pick a ground-truth file.
For the
com-*datasets SNAP ships two:all(every community) andtop5000(the 5000 highest-quality ones, the usual choice for benchmarking). Datasets whose ground truth is a node-label partition have a single file, returned regardless of variant.
- property node_attributes: tuple[str, ...]¶
The column
netgamut.snap.node_table()returns, named after the file.
- netgamut.snap.build_catalog(*, details=True, workers=8, on_progress=None, only=None)[source]¶
Scrape snap.stanford.edu and return a freshly built catalog.
With details the per-dataset pages are fetched too, which is what supplies the statistics tables and download URLs. only restricts the detail pass to the named datasets, which keeps test runs cheap.
- netgamut.snap.catalog(*, reload=False)[source]¶
The SNAP catalog, read from the cache if refreshed, else from the package.
- netgamut.snap.is_cached(name, *, file=None)[source]¶
Whether name is already downloaded, without fetching anything.
- netgamut.snap.load(name, *, file=None, force=False, progress=None, max_bytes=None)[source]¶
Download name if it is not already cached, and return a handle to it.
The download goes to a shared user-level cache (see
cache_dir()), so the second call – in this project or any other – is free.- Parameters:
name (str) – Dataset name, case-insensitive, e.g.
"ca-GrQc".file (str | None) – Which artifact to fetch. Defaults to the dataset’s main edge list; see
info(name).filesfor the alternatives.force (bool) – Re-download even if a cached copy exists.
progress (Callable[[int, int | None], None] | bool | None) –
Truefor a terminal progress bar,Falseto silence it, or a callable receiving(downloaded, total). Defaults to a bar when stderr is a terminal.max_bytes (int | None) – Refuse to download anything larger. Useful as a guard, since some SNAP datasets run to tens of gigabytes.
- Raises:
KeyError – No such dataset, or no such file within it.
LookupError – The dataset has no downloadable file listed.
DownloadError – The transfer failed or exceeded max_bytes.
- netgamut.snap.load_ground_truth(name, *, variant='top5000', force=False, progress=None, max_bytes=None)[source]¶
Download a dataset’s ground-truth communities and return a handle.
Only the Networks with ground-truth communities datasets have these. variant is
"top5000"for the 5000 highest-quality communities, which is the usual benchmark reference, or"all"for the full set:truth = snap.load_ground_truth("com-DBLP") reference = [set(c) for c in truth.communities()]
- Raises:
LookupError – The dataset ships no ground-truth partition. Use
snap.search(ground_truth=True)to see the ones that do.
- netgamut.snap.node_table(network)[source]¶
Read a SNAP dataset’s per-node metadata into the shared shape.
SNAP keeps node attributes in a separate artifact rather than beside the edge list, so this downloads it on the same cache-once terms as
load(). Onlyemail-Eu-corepublishes one today – each person’s department – which is why the column is named after the file rather than from a schema SNAP does not have.Overlapping ground-truth communities are a different thing and stay on
load_ground_truth()andNetwork.communities(): a cover is not a column.- Raises:
LookupError – This dataset ships no node-label artifact.
- netgamut.snap.refresh(*, details=True, workers=8)[source]¶
Re-scrape SNAP and store the result in the user cache.
The refreshed catalog takes precedence over the one bundled with the package, so newly published SNAP datasets become available without waiting for a release. Requires network access.
- netgamut.snap.search(query='', *, category=None, directed=None, max_nodes=None, min_nodes=None, max_edges=None, min_edges=None, ground_truth=None)[source]¶
Find datasets by name, description, category, orientation or size.
For example, every undirected network small enough to load comfortably:
snap.search(directed=False, max_edges=1_000_000)
Or a graded benchmark set with reference partitions to score against:
snap.search(ground_truth=True, max_edges=10_000_000)
Netzschleuder¶
Access to the Netzschleuder network repository (networks.skewed.de).
Netzschleuder is curated by Tiago Peixoto and tags every network by domain, which makes it the source to draw on when a comparison needs to span kinds of system rather than kinds of provenance:
from netgamut import netzschleuder as nz
nz.domains()
# {'Social': 148, 'Informational': 53, 'Biological': 36, ...}
for dataset in nz.search(domain="Biological", max_nodes=1000):
print(dataset)
It complements netgamut.snap, which has no biological, multilayer or
non-road transport networks at all.
Downloads use the site’s CSV bundle, so nothing beyond the standard library is required – graph-tool is not needed despite it being the site’s native format.
Please cite the networks you use; see Dataset.citations.
- class netgamut.netzschleuder.Catalog(datasets, source='https://networks.skewed.de/api', retrieved='', origin='package')[source]¶
An immutable snapshot of the Netzschleuder collection.
- class netgamut.netzschleuder.Dataset(name, tags=(), description='', analyses=<factory>, subgraph_count=0, subgraphs=(), citations=(), upstream_url='')[source]¶
One Netzschleuder network.
A Netzschleuder entry may hold several subgraphs under one name (for example one per year).
subgraphslists them; the statistics here describe the default one.- subgraphs: tuple[str, ...]¶
Up to the first 100 subgraph names, which is all of them for most entries. Longer lists are cut:
openstreetmapalone has over 90,000 subgraphs, which would dominate the shipped catalog. Usesubgraph_names()for all of them.
- property multilayer: bool¶
Whether the site publishes this as a multilayer/multiplex network.
These are the natural test cases for Infomap’s multilayer model, and SNAP has none.
- property multigraph: bool¶
some pair of nodes links more than once.
- Type:
Whether the site tags it a multigraph
- subgraph_names()[source]¶
Every subgraph name, fetched from the API on demand.
Only a preview is vendored, so this needs network access when the entry has more subgraphs than
subgraphsholds.
- property default_subgraph: str¶
The subgraph
netgamut.netzschleuder.load()fetches by default.Downloads are named after the subgraph, not the entry. For most entries the two coincide, but some publish several:
karateholds77and78, the two versions of Zachary’s data.
- files_for(subgraph=None)[source]¶
The download formats the site offers for one subgraph.
The CSV bundle is listed first because it needs nothing beyond the standard library; the others are here for callers that prefer them.
- netgamut.netzschleuder.build_catalog(*, workers=8)[source]¶
Read the whole collection from the Netzschleuder JSON API.
- netgamut.netzschleuder.catalog(*, reload=False)[source]¶
The Netzschleuder catalog, from the cache if refreshed, else the package.
- netgamut.netzschleuder.describe(name)[source]¶
A readable metadata summary for one network, for printing.
- netgamut.netzschleuder.is_cached(name, *, subgraph=None)[source]¶
Whether name is already downloaded, without fetching anything.
- netgamut.netzschleuder.load(name, *, subgraph=None, force=False, progress=None, max_bytes=None)[source]¶
Download name if not already cached, and return a handle to it.
The CSV bundle is fetched and its
edges.csvused as the network, soNetwork.path,.edges()and.to_infomap()work exactly as they do for SNAP data. The bundle’s node and graph properties stay available alongside viaNetwork.extract()andNetwork.nodes().
- netgamut.netzschleuder.node_names(network)[source]¶
Each node’s name from
nodes.csvin the bundleload()fetched, or{}.Network.to_pajek()writes these as the*Verticeslabels. The column is the oneNodeTable.labels()picks.
- netgamut.netzschleuder.node_table(network)[source]¶
Read a Netzschleuder network’s per-node metadata into the shared shape.
The CSV bundle already downloaded for the edge list carries
nodes.csvbeside it, so this costs an extraction and no network access. Every column the site published comes through under its own name:karatelabels each member with their faction undergroups,footballeach team with its conference undervalue. Netzschleuder has no separate ground-truth file, so these columns are what a community-detection run gets scored against – with the usual caveat that an attribute is not ground truth for community structure.Node ids are the row order the bundle uses, which is what
edges.csvrefers to.- Raises:
LookupError – The bundle ships no
nodes.csv.
- netgamut.netzschleuder.refresh(*, workers=8)[source]¶
Re-read the live API and store the result in the user cache.
- netgamut.netzschleuder.search(query='', *, domain=None, subtype=None, directed=None, weighted=None, multilayer=None, bipartite=None, max_nodes=None, min_nodes=None, max_edges=None, min_edges=None)[source]¶
Find networks by domain, subtype, size or structural property.
For a domain-balanced comparison set:
nz.search(domain="Biological", max_nodes=1000)
For Infomap’s multilayer model, which SNAP cannot serve at all:
nz.search(multilayer=True)
KONECT¶
Access to KONECT, the Koblenz Network Collection (konect.cc).
KONECT, curated by Jérôme Kunegis, describes every network in one fixed vocabulary: its format (undirected, directed, bipartite), its edge type (unweighted, multigraph, positive or signed weights), timestamps, and whether it contains loops. The loop check is what no other source here reports, and it is what a benchmark of how community detection treats self-links needs:
from netgamut import konect
konect.search(loops=True, max_nodes=10_000)
konect.search(signed=True)
konect.search(bipartite=True, domain="Economic")
The catalog is KONECT’s downloadable networks minus its per-language Wikimedia series (about a thousand edit, hyperlink and talk networks, one per language edition), of which only the English editions are kept.
A few networks ship per-node attributes – names, sexes, roles – as ent.*
files in their archive. No KONECT page says which, so the catalog build lists
every archive up to 100 MB; search(node_metadata=True) finds those that do,
and Network.nodes() reads them. The larger archives are unlisted, and
answer None.
Netzschleuder imported a third of KONECT, and some of it is SNAP’s. Such a
network is still listed here, but the cross-source netgamut.search(),
netgamut.domains() and netgamut.load() use the other copy; see
duplicates().
Please cite the networks you use; see Dataset.citations.
- class netgamut.konect.Catalog(datasets, source='http://konect.cc/networks/', retrieved='', origin='package', copies=())[source]¶
An immutable snapshot of the KONECT collection.
- copies: tuple[Pair, ...] = ()¶
Every pair of a KONECT network and a network elsewhere that may be its copy, with the verdict the catalog build reached; see
netgamut.konect.duplicates().
- search(query='', *, domain=None, category=None, directed=None, bipartite=None, weighted=None, signed=None, temporal=None, loops=None, node_metadata=None, max_nodes=None, min_nodes=None, max_edges=None, min_edges=None)[source]¶
Filter the catalog by domain, category, size or structural property.
loopsandnode_metadataexclude networks whose answer is unknown – no loop check reported, or an archive too large to have been listed – rather than assuming either answer.
- class netgamut.konect.Dataset(name, title='', description='', category='', format='', edge_type='', loops=None, temporal=False, nodes=None, edges=None, left_nodes=None, node_meaning='', edge_meaning='', statistics=<factory>, citations=(), upstream_url='', archive_bytes=None, archive_modified='', members=None, edge_member='', node_attributes=None)[source]¶
One KONECT network.
- loops: bool | None¶
from KONECT’s own check,
Noneif the page does not say. Bipartite networks cannot have loops.- Type:
Whether any node links to itself
- nodes: int | None¶
n,mas KONECT reports them.mcounts every line, so repeated edges in a multigraph count separately.
- left_nodes: int | None¶
For a bipartite network, how many nodes are on the left. KONECT numbers both sides from 1; the loaded edge list shifts the right side past this.
- archive_bytes: int | None¶
The archive’s size and
Last-Modifiedheader when it was listed. A refresh re-lists an archive only when either has changed.
- members: tuple[str, ...] | None¶
Every file in the archive, by base name.
Nonewhen it was not listed: archives over 100 MB are left unread by the catalog build.
- node_attributes: tuple[str, ...] | None¶
The columns
netgamut.konect.node_table()returns (name,sex), read from the archive’sent.*files when it was listed;()when it ships none,Nonewhen it was not listed.
- property edge_attributes: tuple[str, ...] | None¶
Per-edge files (
person.role), one line per edge;Noneif unlisted.
- property node_metadata: bool | None¶
Whether the archive ships per-node attributes;
Noneif unlisted.
- property duplicate_of: str | None¶
"<source>/<name>"of the same network elsewhere, if there is one.Cross-source queries use that copy instead of this one. See
netgamut.konect.duplicates()for the rules.
- class netgamut.konect.Pair(name, of, verdict, note='')[source]¶
A KONECT network, a network elsewhere it may be a copy of, and the verdict.
- netgamut.konect.build_catalog(*, workers=4, previous=None, list_cap=100000000)[source]¶
Read the downloadable, non-series part of KONECT from the live site.
previous supplies archive listings to reuse; with none, every archive up to list_cap bytes is downloaded once to list it.
list_cap=Noneskips listing entirely. The verdicts on copies are carried over from previous as they stand;scripts/regenerate_konect_catalog.pyjudges them anew.
- netgamut.konect.catalog(*, reload=False)[source]¶
The KONECT catalog, from the cache if refreshed, else the package.
- netgamut.konect.duplicates()[source]¶
KONECT name ->
"<source>/<name>"of the copy cross-source queries use.A KONECT network is a copy only on evidence recorded when the catalog was built, never on its name or its size alone: KONECT’s
epinionsandtwitterare different networks from Netzschleuder’s, andbrunson_southern-womenshares its 10 nodes and 14 edges with an unrelated one. The evidence is either of:imported: Netzschleuder’s entry names the KONECT network as its upstream and holds that one network, not a family of subgraphs.same-edges: the two edge lists were downloaded and found equal, as they stand or up to a relabelling of the nodes.
A pair the build has not compared, usually one above its size cap, stays two separate networks; so do near-copies – a different snapshot, the largest component only.
catalog().copieslists every pair considered, with its verdict and how it was reached.
- netgamut.konect.is_cached(name)[source]¶
Whether name is already downloaded, without fetching anything.
- netgamut.konect.load(name, *, force=False, progress=None, max_bytes=None)[source]¶
Download name if not already cached, and return a handle to it.
The archive is kept as KONECT serves it (
Network.extract()reaches its README and meta files); the edge list is rewritten once beside it so that.path,.edges()and.to_infomap()work as for every other source. Bipartite networks come back with unique ids: right-side nodes start atinfo(name).left_nodes + 1. Signed and rating weights are kept in the file but not handed to Infomap, which accepts only non-negative weights.
- netgamut.konect.node_names(network)[source]¶
Each node’s name from the archive’s
ent.*files, or{}without one.Network.to_pajek()writes these as the*Verticeslabels. The column is the oneNodeTable.labels()picks.
- netgamut.konect.node_table(network)[source]¶
Read a KONECT network’s per-node attributes into the shared shape.
KONECT keeps them as
ent.*files inside the archiveload()already fetched, so this costs an extraction and no network access. The three file shapes it uses – one attribute per file, a profile table, a list of labels – come out as named columns, the onesinfo(name).node_attributeslists. Node ids are the edge list’s, right side of a bipartite network shifted included, so the table andNetwork.edges()agree.Few networks ship such files;
search(node_metadata=True)finds them.- Raises:
LookupError – The archive ships no per-node attributes.
- netgamut.konect.refresh(*, workers=4)[source]¶
Re-read the live site and store the result in the user cache.
Archive listings are carried over from the current catalog while their archives are unchanged, so this downloads an archive only when KONECT has published a new or changed one.
- netgamut.konect.search(query='', *, domain=None, category=None, directed=None, bipartite=None, weighted=None, signed=None, temporal=None, loops=None, node_metadata=None, max_nodes=None, min_nodes=None, max_edges=None, min_edges=None)[source]¶
Find networks by domain, KONECT category, size or structural property.
loops=Truefinds networks with at least one self-link, by KONECT’s own check.node_metadata=Truefinds those whose archive ships per-node attributes, whichnode_table()reads. Networks with no answer – no loop check reported, an archive too large to have been listed – are excluded from either filter.
Open Graph Benchmark¶
Open Graph Benchmark: the graph datasets the GNN literature is benchmarked on.
OGB is small next to the other sources – a dozen single networks – but the metadata
is the best of any of them. Where SNAP gives one labelled dataset and PyG gives
anonymous integers, OGB ships a mapping/ folder that names every code:
ogbn-arxiv node labels decode to cs.NA, cs.LO and the rest, and
nodeidx2paperid.csv.gz resolves each node to a real paper. That is what makes it
worth reading attribute-wise rather than just topologically:
net = netgamut.load("ogbn-arxiv")
nodes = net.nodes()
nodes.columns # ('arxiv_category', 'year', 'paper')
nodes["arxiv_category"] # {0: 'arxiv cs na', ...}
nodes.partition("arxiv_category") # {0: 3, ...}, for AMI/NMI/D
Columns are named the way the source names them – arxiv_category comes from
mapping/labelidx2arxivcategeory.csv.gz, paper from nodeidx2paperid – so a
figure legend can use them as they are. NodeTable’s repr lists what a dataset
carries.
Nothing here needs torch or the ogb package: the archives are plain zips of
csv.gz, and this reads them directly. Only split/*.pt is torch’s, and splits
are a machine-learning concern this package has no use for.
Not everything OGB publishes is loadable. The graphproppred family – 15
collections of thousands of small molecule graphs – is not catalogued at all, since a
Network is one graph. Heterogeneous datasets
(ogbn-mag, ogbl-biokg) and the two stored as data.npz
(ogbn-papers100M, ogbl-vessel) are catalogued but refused, each with its
reason; search(supported=True) filters them out.
Node metadata is not ground truth for community structure. An arXiv subject area
is what an author ticked, not a claim about flow; see netgamut._core.nodes.
- class netgamut.ogb.Catalog(datasets, source='snap-stanford/ogb', retrieved='')[source]¶
An immutable snapshot of the OGB collection.
- class netgamut.ogb.Dataset(name, family='', description='', directed=None, num_classes=None, task_type='', download_name='', version=1, split='', node_files=(), hetero=False, binary=False, nodes=None, edges=None, domains=(), files=(), citations=())[source]¶
One OGB dataset.
- family: str¶
nodeproppredorlinkproppred. Thegraphproppredfamily is not catalogued: those are collections of thousands of small graphs, which the one-graphNetworkmodel has nowhere to put.
- split: str¶
The split OGB recommends. Recorded for provenance; splits are an ML concern and are not read.
- netgamut.ogb.build_catalog(*, timeout=60.0)[source]¶
Read OGB’s
master.csvfiles and build a fresh catalog.
- netgamut.ogb.catalog(*, reload=False)[source]¶
The OGB catalog, read from the vendored snapshot or a refreshed copy.
- netgamut.ogb.datasets(family=None)[source]¶
Every catalogued dataset, optionally restricted to one task family.
- netgamut.ogb.is_cached(name)[source]¶
Whether name is already downloaded, without fetching anything.
- netgamut.ogb.load(name, *, force=False, progress=None, max_bytes=None)[source]¶
Download name if it is not already cached, and return a handle to it.
The archive is fetched whole – OGB ships one zip per dataset – and the handle points at its
raw/edge.csv.gz, so.edges(),.to_networkx()and.nodes()all work as they do for every other source.Sizes are real:
ogbl-ddiis 46 MB,ogbn-arxiv83 MB,ogbn-productsover 1 GB. Pass max_bytes as a guard.- Raises:
KeyError – No such dataset.
LookupError – The dataset is catalogued but not loadable – heterogeneous, or stored as
data.npz.DownloadError – The transfer failed or exceeded max_bytes.
- netgamut.ogb.node_table(network)[source]¶
Read an OGB dataset’s per-node metadata into the shared shape.
Everything comes out of the archive already downloaded for the edge list: the label column from
raw/node-label.csv.gz, its names frommapping/labelidx2*.csv.gzwhere OGB provides them, any extra per-node columns the catalog lists (node_year,node_species), and the entity mapping frommapping/nodeidx2*.csv.gz.A multi-task dataset labels each node with a vector rather than a class –
ogbn-proteinscarries 112 binary indicators – so those arrive aslabel_0 .. label_111. Its meaningful categorical attribute isspecies.- Raises:
LookupError – The archive carries no per-node files at all.
PyTorch Geometric¶
PyTorch Geometric datasets, materialised once into plain files.
PyG is where most of the graph-learning literature’s networks live. This catalogues the
single-graph ones – the collections of thousands of molecules or meshes are left out,
since a Network is one graph – and loads them through
the same call as every other source:
net = netgamut.load("Planetoid/Cora")
nodes = net.nodes()
nodes["label"] # {0: 3, ...}
nodes.partition("label") # {0: 3, ...}, for AMI/NMI/D
The torch dependency is a one-time cost. The first load needs the [pyg] extra:
PyG downloads its raw files, and this converts the result to edges.csv.gz and
nodes.csv.gz in the cache. Every load after that reads those two files and never
imports torch. The materialised copy is a plain, archivable network like any other.
What is deliberately not written: the dense node-feature matrix (Cora’s 1,433
bag-of-words dimensions), which is a learning input rather than network metadata, and
the train/validation/test masks, which are a benchmarking protocol. Use
from_pyg() on a Data object when you want those.
PyG labels are anonymous. Data.y is an integer tensor and there is no
class_names anywhere in its API, so nodes.categories("label") returns the codes
themselves. What they mean lives in each dataset’s originating paper – unlike
netgamut.ogb, which ships the names. And node metadata is not ground truth for
community structure either way; see netgamut._core.nodes.
- class netgamut.pyg.Catalog(datasets, source='pyg-team/pytorch_geometric', retrieved='', omitted=None)[source]¶
An immutable snapshot of the PyG collection.
- class netgamut.pyg.Dataset(name, cls='', dataset='', description='', nodes=None, edges=None, features=None, num_classes=None, directed=None, domains=(), paper='', load_error='')[source]¶
One named PyG dataset: a class plus, where the class spans several, a name.
- nodes: int | None¶
PyG’s own figures, from the class docstring.
Nonewhere it publishes none: the cheatsheet is explicitly a work in progress.
- directed: bool | None¶
PyG stores undirected graphs as both directions; nothing in the docstrings says which are directed, so this stays unknown rather than being guessed.
- load_error: str¶
Why this dataset cannot be constructed, when a measuring pass found out. Only structural failures are recorded – a constructor argument we do not supply, or a name PyG refuses in the mode we use. Network errors and missing optional dependencies are environmental and deliberately not baked in, since they say nothing about the dataset.
- property files: tuple[Any, ...]¶
PyG downloads its own raw files; nothing is listed here to fetch directly.
- netgamut.pyg.build_catalog(*, timeout=60.0)[source]¶
Parse PyG’s dataset classes into a catalog.
Classes with a statistics table are catalogued from it. Classes without one are catalogued only when
DOMAIN_MAPvouches for them, using the dataset names in theirname (str):argument documentation; their sizes stay unknown. Everything else is recorded inCatalog.omitted.
- netgamut.pyg.catalog(*, reload=False)[source]¶
The PyG catalog, read from the vendored snapshot or a refreshed copy.
- netgamut.pyg.datasets(cls=None)[source]¶
Every catalogued dataset, optionally restricted to one PyG class.
- netgamut.pyg.from_pyg(data, *, directed=None)[source]¶
Convert a PyG
Dataobject you already have into a graph and a node table.The escape hatch from the catalog: anything PyG can produce – a dataset this package does not list, one graph out of a collection, a transformed object – can still enter the same API. Needs networkx, and torch only because data is already a torch object.
Returns
(graph, NodeTable).
- netgamut.pyg.is_cached(name)[source]¶
Whether name has been materialised already, without importing torch.
- netgamut.pyg.load(name, *, force=False, progress=None)[source]¶
Materialise name if it is not already cached, and return a handle to it.
The first call needs the
[pyg]extra and runs PyG’s own downloader; it writesedges.csv.gzandnodes.csv.gzinto the cache. Later calls read those and import nothing heavy.- Raises:
KeyError – No such dataset in the catalog.
LookupError – A measuring pass established this one cannot be constructed.
ImportError – First load, and PyTorch Geometric is not installed.
LookupError – The installed PyG has no such class, or it turned out to hold several graphs.
- netgamut.pyg.node_table(network)[source]¶
Read a materialised PyG dataset’s per-node metadata into the shared shape.
Reads the
nodes.csv.gzwritten at materialisation, so it needs no torch. PyG names no categories, socategories(col)returns the integer codes; the meaning of a code lives in the dataset’s originating paper.- Raises:
LookupError – The materialised copy carries no node columns – some PyG datasets have no
yat all.
- netgamut.pyg.omitted()[source]¶
Classes PyG offers that are not catalogued, and why.
Worth reading before concluding a dataset is missing: most entries are collections of many graphs, and the rest are classes PyG has not written a statistics table for, which its cheatsheet describes as a work in progress.
- netgamut.pyg.refresh(*, timeout=60.0)[source]¶
Rebuild from PyG’s source into the user cache and return the new catalog.
- netgamut.pyg.search(query=None, *, domain=None, max_nodes=None, cls=None, directed=None, supported=None)[source]¶
Filter the catalog. query matches the name and description.
max_nodesmatches only datasets whose size PyG publishes, anddirectedonly those a measuring pass has loaded: PyG states neither, so an unmeasured dataset is excluded rather than assumed to be anything.supported=Falselists the ones a pass found could not be constructed at all.
Path data¶
Path datasets and the memory networks built from them.
Flows in real systems remember where they came from: a passenger’s next flight depends on the itinerary so far, a reader’s next click on the trail behind it. This source curates datasets whose unit of observation is a whole path, and pairs them with the conversion into Infomap state networks:
from netgamut import paths
data = paths.load("wikispeedia") # 9.5 MB, 51,318 navigation trails
memory = data.to_state_network(order=2)
memory.write("wikispeedia_states.net") # *Vertices / *States / *Links
im = memory.to_infomap(silent=True)
im.run()
With order 2, every observed trigram i -> j -> k adds a link from state
(i, j) to state (j, k) – being at j, remembering i. Order 1
gives the plain directed network from the same paths, so first-order versus
second-order comparisons are a one-argument change.
The flight itineraries come in two views of one download – db1b-coupon
between airports, db1b-city between metropolitan areas – and carry the
side tables a plot needs:
flights = paths.load("db1b-coupon")
where = flights.coordinates() # {"LAX": (33.94, -118.41), ...}
cities = paths.load("db1b-city") # same archive, no second download
cities.names()["31703"] # 'New York City, NY (Metropolitan Area)'
The collection is curated here rather than scraped from a site, which is what guarantees every entry is downloadable without registration and ships with a parser that knows its quirks (Wikispeedia’s back clicks, DB1B’s scattered coupon rows).
Please cite the datasets you use; see Dataset.citations.
- class netgamut.paths.Catalog(datasets, source='curated', retrieved='', origin='package')[source]¶
An immutable snapshot of the curated path datasets.
- class netgamut.paths.Dataset(name, description='', parser='', files=(), metadata=(), shares_download_with='', url='', url_template='', default_year=None, default_quarter=None, nodes=None, num_paths=None, citations=(), license='')[source]¶
One curated path dataset.
- metadata: tuple[tuple[str, DataFile], ...]¶
Small side tables fetched on demand rather than with the download, as
(role, file)pairs. Roles are what the handle’s accessors look up:"coordinates"forPathData.coordinates(),"names"forPathData.names().
Another entry whose cache directory holds this one’s downloads. Two views of one archive –
db1b-cityreads the same coupon file asdb1b-coupon– then share a single copy of it.
- property edges: int | None¶
the link count depends on the chosen Markov order.
- Type:
Unknown up front
- property node_attributes: tuple[str, ...]¶
the paths source publishes no per-node metadata.
- Type:
None
- property cache_name: str¶
The dataset directory in the cache; another entry’s when downloads are shared.
- metadata_file(role)[source]¶
The side table filling role, e.g.
"coordinates".- Raises:
LookupError – This dataset ships no such table.
- file_for(*, year=None, quarter=None)[source]¶
The artifact to download, resolving year/quarter for quarterly data.
- Raises:
ValueError – year/quarter passed for a single-collection dataset, or out of range for a quarterly one.
- class netgamut.paths.PathData(dataset, file, archive_path, member=None, source='', edge_file=None, columns=Columns(weight=None, time=None, layer=None, names=()), parser='')[source]¶
A downloaded path dataset on local disk.
Everything a
Networkoffers, plus the stream of trajectories and the conversion into a memory network:data = paths.load("wikispeedia") for nodes, weight in data.paths(): ... memory = data.to_state_network(order=2)
- paths(*, max_paths=None, max_nodes=None)[source]¶
Stream every path as
(nodes, weight).Weights count observations: passengers for flight itineraries, one per trail for navigation data. Streaming keeps memory flat; parsing details – and any one-time derivation work – live with the dataset’s parser, see
netgamut.paths._parsers.For demonstrations and small benchmarks the stream can be subset, deterministically:
- Parameters:
max_paths (int | None) – Keep only the first max_paths paths, in file order.
max_nodes (int | None) – Keep only the max_nodes most-visited physical nodes (visits weighted by path weight, ties broken by first appearance), applied after max_paths. Paths are cut at removed nodes rather than bridged across them – a trajectory
a -> x -> bwithxremoved yields noa -> btransition – and the surviving segments of at least two nodes are streamed as separate paths. Costs one extra pass over the (already limited) stream to rank the nodes.
- to_state_network(*, order=2, max_paths=None, max_nodes=None)[source]¶
Build the memory network of the given Markov order from the paths.
With the default order 2, every observed trigram
i -> j -> kcontributes a link from state(i, j)to state(j, k); state(i, j)is physical nodejremembering the arrival fromi.max_paths and max_nodes subset the trajectories first – see
paths()– which is the handle for demo-sized or benchmark-sized networks from a large dataset.
- coordinates()[source]¶
Node label ->
(latitude, longitude), for plotting.Read from the dataset’s
"coordinates"table, fetched once into the same cache directory as the download. For the DB1B airport itineraries this is the OpenFlights airports table, keyed by three-letter IATA code, socoordinates().get(airport)positions a node – a few small airfields in DB1B have no OpenFlights entry and come backNone.- Raises:
LookupError – This dataset ships no coordinates table.
- names()[source]¶
Node label -> human-readable name, from the dataset’s
"names"table.The DB1B city itineraries are labelled by BTS city-market id; this maps
"31703"to"New York City, NY (Metropolitan Area)"via the BTSL_CITY_MARKET_IDlookup, fetched once next to the download.- Raises:
LookupError – This dataset ships no names table.
- edges(*, cast=None)[source]¶
Stream the first-order projection: consecutive pairs of every path.
This is what makes
to_networkx()work unchanged on path data. Path weights are not repeated per pair; useto_state_network()withorder=1when weights matter.
- to_pajek(directory, *, name=None, directed=None, names=None, order=1)[source]¶
Write the network the paths make, weights kept, and return its path.
Order 1 by default, matching
to_infomap(): a directed Pajek file of the first-order network.order=2and up writes the memory network in Infomap’s states format; seeStateNetwork.to_pajek(). directed is accepted for signature parity withNetwork.to_pajek()and must not beFalse: transitions along a path are directed. names is accepted for the same reason and must be empty: path nodes are named by their own labels.
- to_link_list(directory, *, name=None, directed=None)[source]¶
Write the first-order network of the paths as a link list.
Node labels are numbered from 1 in first-seen order – a link list cannot hold names – so prefer
to_pajek()unless the labels are meaningless anyway.
- to_infomap(*, directed=None, order=1, **kwargs)[source]¶
Build an
infomap.Infomapinstance from the paths, weights kept.Defaults to the first-order network, matching what
to_infomapmeans everywhere else in this package; passorder=2for the memory network, or callto_state_network()first to keep the intermediate representation.
- netgamut.paths.catalog(*, reload=False)[source]¶
The curated catalog of path datasets, vendored inside the package.
- netgamut.paths.is_cached(name, *, year=None, quarter=None)[source]¶
Whether name is already downloaded, without fetching anything.
- netgamut.paths.load(name, *, year=None, quarter=None, force=False, progress=None, max_bytes=None)[source]¶
Download name if not already cached, and return a handle to it.
The handle adds
paths()andto_state_network()on top of the usualNetworkinterface, pluscoordinates()andnames()for the entries that ship those side tables. Entries that are views of one archive (db1b-cityreadsdb1b-coupon’s file) share its cache directory, so the download happens once.
AHORN¶
AHORN: the higher-order datasets the other four sources cannot express.
The Aachen Higher-Order Repository of Networks curates datasets whose unit of observation is a set of nodes rather than a pair: hypergraphs, simplicial complexes, one combinatorial complex. Three authors on one paper is one interaction here, not three edges, and no edge list can say which of the two it was. That is the gap this source fills – SNAP, Netzschleuder, OGB and PyG all publish dyadic graphs, and AHORN’s own survey files them under “higher-order structure only implicit”:
from netgamut import ahorn
ahorn.search(network_type="simplicial-complex", max_nodes=1000)
h = ahorn.load("algebra-questions")
h.metadata # {'name': ..., '_format-version': '0.2'}
for members, attributes in h.interactions(cast=int):
... # ((2, 12, 20), {'category': 'lie-algebras'})
im = h.to_infomap() # bipartite, so interaction sizes survive
im.run()
load() returns a Hypergraph, not a
Network: flattening is a choice you make with to_networkx(projection=...)
or to_network(...), not something this does behind your back.
Revisions are pinnable, and only here. AHORN publishes each dataset as
versioned Zenodo records, so load(name, revision=1) keeps returning the same
bytes after a correction lands. Without it you get the latest, which is
convenient and not reproducible – algebra-questions revision 2 renamed an
attribute its revision 1 called genre.
Citation is deliberately two-part. Cite the original source publication – the
BibTeX in Dataset.citations – for the data, and the AHORN paper for the
repository; ahorn.CITATION carries the latter. Licences are the upstream
source’s, recorded on the record and granting nothing beyond what that source
already allowed.
- class netgamut.ahorn.Catalog(datasets, source='netsci-rwth/ahorn', retrieved='')[source]¶
An immutable snapshot of the AHORN collection.
- class netgamut.ahorn.Dataset(name, title='', description='', network_types=(), tags=(), domains=(), nodes=None, interactions=None, max_interaction_size=None, mean_interaction_size=None, max_node_degree=None, mean_node_degree=None, temporal=False, weighted=False, multi_network=False, source_url='', license='', license_url='', citations=(), parent='', related=(), revisions=(), directed=False)[source]¶
One AHORN dataset.
- network_types: tuple[str, ...]¶
The formalisms AHORN suggests reading it as –
hypergraph,simplicial-complex,combinatorial-complex,graph. Suggestions, in AHORN’s words, not claims to completeness.
- tags: tuple[str, ...]¶
domain: reviews,source: benson,network-type: temporal,metadata: node classes.- Type:
AHORN’s own namespaced tags, verbatim
- interactions: int | None¶
Hyperedges, simplices, cells – whatever this dataset’s formalism calls them. The count AHORN reports as
num-interactions.
- license: str¶
An SPDX identifier, or
unprovided-reuse-encouragedwhere the upstream source stated none. A dataset page documents the upstream conditions; it does not grant rights beyond them.
- citations: tuple[str, ...]¶
BibTeX for the original source publication. Cite this, plus the Zenodo record when the exact version matters – AHORN keeps repository citation and dataset citation deliberately separate.
- parent: str¶
The slug this was derived from, for the cuisine- and genre-specific children that share a converter with their parent.
- directed: bool | None¶
Interactions are unordered sets, so there is nothing to be directed about. Present because every source’s record must answer this.
- property edges: int | None¶
The interaction count, under the name cross-source
search()filters on.An interaction is not an edge. This exists so
max_edges=keeps meaning “not bigger than this” across all sources; readinteractionswhen you mean it.
- revision(number=None)[source]¶
One revision by number, or the latest when number is
None.- Raises:
LookupError – No such revision, or the dataset publishes none.
- class netgamut.ahorn.Revision(number, url, hif_url='', size=None, changelog=())[source]¶
One published, citable version of a dataset.
Revisions are archived as Zenodo records, so a pinned one keeps returning the same bytes after AHORN corrects or re-derives the dataset.
changelogis what changed relative to the revision before it.
- netgamut.ahorn.build_catalog(*, timeout=120.0, archive=None, sizes=True)[source]¶
Read AHORN’s dataset frontmatter and build a fresh catalog.
Pass archive to parse a repository tarball you already have; otherwise it is fetched from GitHub. sizes additionally reads the site index for the per-revision file sizes the frontmatter omits.
- Raises:
ImportError –
pyyamlis not installed. Only building needs it; the vendored catalog reads without it.DownloadError – The repository archive could not be fetched.
- netgamut.ahorn.catalog(*, reload=False)[source]¶
The AHORN catalog, read from the vendored snapshot or a refreshed copy.
- netgamut.ahorn.datasets(network_type=None)[source]¶
Every catalogued dataset, optionally restricted to one formalism.
The formalisms overlap: most hypergraphs also admit a simplicial reading, so a dataset can appear under both.
- netgamut.ahorn.is_cached(name, *, revision=None)[source]¶
Whether name is already downloaded, without fetching anything.
- netgamut.ahorn.load(name, *, revision=None, force=False, progress=None, max_bytes=None)[source]¶
Download name if it is not already cached, and return a handle to it.
Without revision you get the latest published one, which is what the website links. Pass a number to pin it: revisions are separate Zenodo records, so a pinned call keeps returning the same bytes and a benchmark stays reproducible across a correction.
Revisions are cached side by side under the dataset’s directory, so pinning one does not evict another.
- Raises:
KeyError – No such dataset.
LookupError – No such revision, or the dataset publishes none.
DownloadError – The transfer failed or exceeded max_bytes.
- netgamut.ahorn.node_table(handle)[source]¶
Read an AHORN dataset’s per-node metadata into the shared shape.
Node attributes live on the explicit node lines of the dataset file, so this reads the file already downloaded and needs nothing else. Most datasets declare no nodes at all – their nodes are implied by the interactions – and those raise rather than returning an empty table.
Interaction attributes are not node attributes and are not here; stream them with
Hypergraph.interactions(), or take one column of them withHypergraph.interaction_labels(...).- Raises:
LookupError – The dataset declares no nodes explicitly.
- netgamut.ahorn.refresh(*, timeout=120.0)[source]¶
Rebuild the catalog into the user cache and return it.
Needs
pyyaml; seebuild_catalog().
- netgamut.ahorn.search(query=None, *, domain=None, network_type=None, tag=None, temporal=None, parents_only=None, max_nodes=None, min_nodes=None, max_interactions=None, min_interactions=None, max_interaction_size=None)[source]¶
Filter the catalog. query matches the slug, title and description.
tag matches AHORN’s own namespaced tags as substrings, so
tag="benson"finds everything derived from the Benson collection andtag="node classes"everything carrying a node labelling.parents_only drops the derived children – the 20 cuisines split out of
cooking, the genre splits of the review datasets – which otherwise dominate any unfiltered listing. max_interaction_size is the guard worth setting before a clique projection: one interaction of k members becomes k(k-1)/2 edges.