All the URLs we extract will need to be stored in a large data structure.
The way I see it is similar to a linked list but with multiple dimensions.
The object will probably have two properties :
- The actual URL string.
- A collection of these objects that this URL connects to, i.e. the valid URLs that were found on this page.
With these properties we can tell what URLs are connected and in which directions.
All the URLs we extract will need to be stored in a large data structure.
The way I see it is similar to a linked list but with multiple dimensions.
The object will probably have two properties :
With these properties we can tell what URLs are connected and in which directions.