Index of /files/Udemy 02/Udemy - Web Scraping in Nodejs & JavaScript 2024-2/11. Robots.txt parsing/

../
1. What is robots.txt.mp4                          22-Feb-2026 22:35            11341346
1. What is robots.txt.vtt                          22-Feb-2026 22:34                4511
2. Initialising project.mp4                        22-Feb-2026 22:35             6436942
2. Initialising project.vtt                        22-Feb-2026 22:35                1892
3. Example of usage robots-parser.mp4              22-Feb-2026 22:35            20025362
3. Example of usage robots-parser.vtt              22-Feb-2026 22:35                6805
4. Parsing robots.txt from a real site.mp4         22-Feb-2026 22:35            75389981
4. Parsing robots.txt from a real site.vtt         22-Feb-2026 22:35               10021