Welcome back to the course.
Today I'm going to show you how to set up and use AP files your MCP server.
And honestly, I think this is going to completely change how you approach data scraping.
Basically, you will be able to write a single prompt and have AI pull live data from practically anywhere,
including Instagram, LinkedIn, Google Maps, you name it.
Just tell it exactly what you want to scrape and it handles everything for you.
Now, the best part about this setup is there is no coding involved.
So you give it a simple instruction and it goes out and gets the data in exactly the format you need.
So we are talking about getting this running just in just a few minutes.
You just ask AI to do it and it knows its tools to use.
How to structure the request and delivers organized data back to you.
What makes this powerful is that it can combine multiple data sources in a single request.
So you could ask it to scrape Instagram posts about a specific topic, then find related LinkedIn profiles
and cross-reference that with Google Maps business data all in one go.
Now, the traditional way of doing this would require multiple tools and separate API keys for each
of these services, and a lot of manual setup.
With this setup, it's all handled automatically through one interface.
Let me show you exactly how this works, and we will start with the basics of what Apify is and why
this integration is such a game changer.
Each scraper has its own pricing structure.
For example, this scraper.
Cost $6 for every thousand results you extract.
And the good news is you can test these scrapers directly on the website for free.
For example, if you take this URL and click start, it's going to run the operation and start pulling
product data from Amazon's website.
As you can see, the actor is getting your data.
It usually takes around one minute to start seeing results.
So you can see it's working through different products right now and we are getting structured data
back.
We got product titles, brand names and the actual URLs for the for each product.
So this is exactly the kind of data you will get from most scrapers.
Clean, organized information that you can feed into whatever language model you are using for analysis,
or any other purpose you have in mind.
But for example, if you wanted to use Google Maps to extract data from thousands of locations and businesses.
Things like reviews, reviews, reviewer details, images, contact info, opening hours, location,
data prices and so on.
You would need to pay per event, which means you are charged for each business location you scrape
data from.
So when you click on pricing, you will see it's paid per event so you are not charged for the Apify
platform usage, just a fixed price for specific events.
So as you can see, the base cost is 0.004 per place scraped.
And there are additional costs for extras like 0.001 for filter applied, 0.002 for additional price
details.
So you pay for exactly what you extract.
So the pricing varies depending on what you are extracting and how much detail you need.
But the key thing is that you are you're getting professional grade data extraction without having to
build these tools yourself.
And some of the scrapers are free.
What's impressive is the scale here.
There are over 5000 actors, so different scrapers available across categories like social media agents,
lead generation.
So it's really comprehensive for almost any data extraction task you can think of.
And the beauty of this new Apify MCP server is that it lets you use all these different scrapers through
one credential, instead of managing separate keys for each service.
So just a few months ago, if you wanted to call different services, you would typically need to send
requests with API keys.
So each key was your credential that opened access to that specific service.
So you'd have one key for Instagram scraping, another for LinkedIn, another for Google Maps, and
so on.
So each key gives you access to one specific type of data or service by using MCP servers.
One key or one set of credentials can access multiple services at the same time.
So before you'd have to create separate credentials to scrape Instagram.
Then another set for Amazon.
Another for X, another for Facebook.
Now you can use the same credential for all of them.
So on this diagram you can see that on the left you have MCP clients.
So these are applications like cloud desktop cursor or other AI tools.
In the middle you have the API MCP server which acts as a central hub.
And on the right you can see all the different platforms it can connect to.
So Instagram, TikTok X and thousands of other services and platforms.
The key difference is that instead of each client needing separate connection to each platform.
Everything goes through this central MCP server.
So when you make a request in Cloud Desktop, it can automatically access Instagram for posts, TikTok
for videos, X for tweets, and all through that single connection.
So with MCP servers, you can make one request that has different conditions.
And it's intelligent enough to figure out which services to use and in what order.
It automatically handles all the different platforms and executes the actions that makes the most sense.
So for actually using MCP servers, I'm working with Cloud Desktop here.
So thanks to that I can have conversations with large language model like Cloud Sonnet or Cloud Opus
that has access to all these different scrapers.
You have been looking at.
So when I click on this drop down, you can see that here I have access to a file server and I have
access to all these different options.
And what's really cool is that some of these tools will actually help me search Apify to discover what
services exist for whatever request I have.
So if you haven't set up a specific scraper yet, this system can help me locate the best ones on the
platform.
It can find scrapers based on reviews and pricing.
So setting this up is actually very simple.
We need to edit a configuration file in Cloud desktop app as we did in our previous lessons.
So go to Cloud Desktop app and in the top menu click on settings.
Then Developer settings.
So when I click on Edit config it opens up a folder.
And then when I click on this file.
So cloud desktop config JSON.
It shows me all the credentials I currently have set up.
Now if you haven't used Cloud desktop app before, this will be a blank file which is actually easy
to work with.
In my case, you can see I have a different services already configured.
So when I scroll down you can see that I have an SMTP server with some configuration details and the
actual service name.
Apify SMTP server.
So below that is the list of different scrapers that I want to have available out of the box.
So I can use YouTube comment scraper TikTok scraper, Instagram scraper, YouTube transcript scraper,
tweet scraper, Facebook ad scraper, and so on.
So adding another scraper is really simple.
Let's say I want to add something new.
I can go back to Apify store.
First of all, if you are new to Apify, please use the link from the resources section of this lesson.
Create your account and sign in by using this link.
You will get $5 in free credits to use.
Then go to Shopify store.
And here you can browse by category so you can find scrapers for social media, for e-commerce, SEO
tools and so on.
Or you can search by specific platform, for example, LinkedIn.
Let's say I want to scrape LinkedIn profiles with emails.
You can also set up filters for example popular.
So I'm going to select this one.
Obviously we want to double check that.
It's going to give you the data you are expecting.
So in order to do this, you can run this and see what kind of results you get.
So assuming you find it useful, you would need to look at the price too.
So in that case.
So for this particular scraper you would need to pay $10 for 1000 results.
I'm not going to open this LinkedIn profile for privacy reasons.
Let's say I want to scrape Facebook ads.
So I'm going to search for actor type Facebook.
And here we go.
You can add this scraper to analyze your competitors advertising strategies or identify trending products
in niche.
So you can find a lot of scrapers too.
And now in order to add the scraper.
So this tool to our MCP server, all we need to do is literally copying this name and adding to our
file.
So we are going to click on this copy button and paste it back into our configuration file.
So the process is very simple.
In our actor MCP server connection and the actors, just after any of the actors add a new name of the
scraper.
When I check my cloud desktop app.
And click on this drop down.
So my tools library.
You can see that in the Apify MCP SMTP server.
I have access to 28 tools.
And now when I restart the cloud and add the name of the new scraper and comma.
And then save the file.
Now when I click on this drop down to see my tools library.
You can see that in the SMTP server.
Now I've got access to 29 tools not 28.
And when I type Facebook I have access to this specific scraper.
So apify Facebook ad scraper.
So this is scraper we just added.
If for whatever reason you want to remove scraper you can either toggle it off as simple as that, or
go back to that configuration file and delete it from the list.
Just remember to remove the comma as well.
Now let's actually see our scrapers in action.
So first let me test the Google Maps scraper.
So I will ask it to find ten digital marketing agencies in San Francisco.
With their.
Contact details.
Ratings and service areas for potential partnership.
And hit send.
Actually I'm going to add use Google maps.
To find I'm being specific about using Google Maps because this could be this could technically be done
with web search, but Google Maps give us more accurate and more richer business data.
So by default, cloud language model could use web search instead of using this tool in our SMTP server.
So I'm going to hit send.
So if you find it's not using the tool you want you can explicit and say use for example Apify Google
Maps scraper tool.
And it will prioritize that specific tool.
Or you can just toggle off the default web search tool in Cloud desktop app.
So now cloud asks us about the permission to use this tool.
So I'm going to allow always.
Here you can see the request.
Let's wait a moment.
Now it's using Compass Cloud Google places tool.
So the tool we have connected to our MCP server.
Again you can find it here.
Now they're asking us about using another tool.
I'm going to allow always.
I'll help you find digital marketing agency in San Francisco using Google Maps data.
Let me search for agencies and extract the contact details, ratings and service information.
Now let me get more details from the complete data set to give you comprehensive information about all
agencies.
Perfect.
I found ten digital marketing agencies in San Francisco with comprehensive details.
Here is a detailed overview of each agency for potential partnership consideration.
Top digital marketing agencies in San Francisco.
We have a rating of address, website, email or paying our services specialties.
Awesome.
For partnership consideration.
Top performers by reviews.
Service areas.
Common specialties.
Contact recommendations.
So now we can actually just check one of the agency.
Regret.
For the next test.
Let me try TikTok scraper.
Now you search for.
TikTok.
Videos.
With AI.
AI ML.
Scraper.
The hashtag.
Could you find me?
Pardon me.
Five viral TikTok videos with that hashtag.
Okay.
Let's see.
So this is a trending hashtag where people use AI to create viral ASMR content with with Google's VR
three sedans or link two.
So text to video models.
Right.
Now it's using clockworks TikTok scraper.
So I limit it to five videos and the system goes through multiple accounts, view counts like comments
and so on.
Let me get more results to find better viral videos with higher engagement.
I'll help you find viral TikTok videos.
Let me search for TikTok videos using relevant hashtags and search terms.
Let me get more results to find better viral videos with higher engagement.
Perfect!
I found some great viral TikTok videos with with eye ASMR content based on the search results.
Here are the five most viral TikTok videos with the with this hashtag.
Okay.
So as you can see, each results includes direct links to watch the videos.
We've got likes, shares, views, comments, description, duration and why it's vital.
All right let's check this one.
Okay.
And this one.
All right.
So this kind of social media system would normally require specialized tools or manual research.
Now for Instagram I will ask you to use.
Instagram scraper.
To analyze.
The last 20 posts.
From this account.
I'm going to provide it with a the name of this account.
Interesting AF.
So we want to scrape this account.
And also we want it to identify.
The most.
Engaging posts.
The most engaging.
Topics.
And posting patterns.
And hit send.
So it automatically selects the Instagram scraper and receives the data.
I'll help you analyze the Instagram account using Instagram Scraper to examine the last 20 posts and
identify engaging posts and topics and posting patterns.
So it's using Shopify Instagram Scraper.
In the request, we pass the URL and we also limited the results to 20.
Right thinking now.
Scrape Instagram data.
Analyze engagement and posting patterns.
Now let me get the complete data set to analyze all the posts and their engagement patterns.
Now it's using get dataset items to.
Analyze Instagram posts.
Perfect!
I successfully scraped 19 posts from this account.
Perfect.
Let's see the results.
A Gallery engagement report.
The analysis of the last 19 posts from this strategy focused on AI generated content.
Top performing posts by engagement.
We got the dates.
Captions.
Comments.
Likes.
Most engaging content categories.
Posting patterns and strategy.
Content source strategy.
Key success factors.
Even recommendations for optimization.
So engagement rate analysis.
Average likes over 2000 likes.
Average comments.
Video performance interactive content.
Look at that.
It's very comprehensive.
Most engaging topics.
Content strategy breakdown top performing patterns.
So we received what we asked for.
That's amazing.
So this kind of competitive social media analysis would normally require A lot of work and data collection.
Now let me test the YouTube transcript scraper.
So I'll ask it to extract the full transcript.
From this video.
And I'm going to provide the link to.
The last video published by Y-combinator.
So you want it to scrape the transcript from this video.
So the Andrej Karpathy Talk software is changing again.
And create.
A structured.
Summary.
With the main steps and key.
Takeaways organized by topic.
They presented.
As a visual.
Mind map.
By the way, I strongly recommend you watching this video.
Andrej Karpathy is a genius, actually.
For this example, let's use the extended thinking feature.
So it will improve the scraping process.
And now you can see the decision making process.
So it determines that it needs the YouTube transcript scraper from Apify and automatically structures
the API request.
Something you would normally have to configure manually in tools like N810 or Make.com.
So I will help you extract the transcript from that YouTube video and create a structured summary with
a visual mind map.
Let me start by extracting the transcript.
As you can see, it's using YouTube transcript scraper.
Now let me get the full transcript since it was truncated.
Right.
Perfect.
I have extracted the full transcript for Andrej Karpathy's talk.
We have the main thesis topics and takeaways.
We've got our alarms and operating systems.
Now it's creating the mind map for us.
Let's wait for our mind map.
All right.
That's impressive.
That's amazing.
So as you can see the complete transcript organized summaries.
And now when I click publish.
Link.
And pass it to my browser.
I have this beautiful mind map from Andrej Karpathy talk.
Software is changing again.
So we have organized summaries and structured breakdowns.
So I asked it to create a visual mind map showing how all the concepts connect to each other.
And from one URL and one request, we get this comprehensive analysis presented in the mind map that
would typically require multiple separate tools and automations.
That's just mind blowing.
I really like it.
So we've got six main topics everyone as a programmers vibe coding with natural language, building
for agents, right.
Software paradigms, llms as operating systems.
That's an interesting topic.
LLM psychology and partial autonomy apps.
I really enjoyed this talk and I highly recommend you watching it.
So here we also have a summary of the process and the topics covered.
Main takeaways.
So in the text format now let's test something for recruitment.
So I'll ask it to.
Use LinkedIn.
Scraper.
Find.
A automation specialist.
In the Boston area.
Who have experience?
With tools like N810.
Maybe to find five automation specialists.
And let's see.
If I can click the tools.
It's asking for permission to use the scraper.
I'm going to allow it once.
I help you find a automation specialist in the Boston area with experience.
All right.
Thinking Now it's asking for permission again.
So I'm going to allow always this time.
Let me try a broader search with more general automation and AI related terms to find relevant professionals.
Okay.
Let me try simple search approach with just basic automation terms and see if we get some results.
Great.
I found some relevant profiles.
Let me get more details from the search results and do another search to find more automation specialists.
Now let me run another search to find more automation specialists with different search terms.
Alright, we have to be patient here.
Perfect.
Now I have some excellent results from the second search.
Let me get full details from this latest search.
All right.
So we have some top AI automation specialists in Boston area.
All right.
Great.
We have title company location, skills experience link to LinkedIn profile.
And why is it perfect for ten.
Best matches for experience.
All right.
Awesome.
Now, um, finally I will combine data scraping with analysis.
So I'm going to ask.
Use Twitter.
With the scraper.
To scrape the last 50.
Or maybe 20 viral posts.
from.
Tech tech industry industry leader and analyze.
The teams.
Predict the next trends.
Let me start scraping posts from OpenAI CEO, since it has been very influential in AI trends.
If you would prefer a different leader, Elon Musk.
Jensen Huang.
Satya Nadella.
Let me know.
You can see what he's doing now step by step.
Yes.
So the results include predictions based on the based on patterns in the content.
So based on the viral tweets from tech leaders such as Sam Altman.
Satya Nadella.
Elon Musk.
Her team analysis.
Trend predictions for 2025 2026.
Autonomous agents.
Mainstream consumer brain computer interface.
Quantum AI.
Hybrid systems.
Retail.
Multimodal AI.
Native infrastructure.
Personalized AI companions.
Predicted timeline.
Actionable insights.
Investment opportunities.
Business strategy.
Technology focus.
Implementation guide.
How to run this analysis.
So you got this nice tech trend analysis dashboard to analyze viral posts from tech industry leaders
to predict future trends.
But as you can see here, it didn't use that Twitter scraper.
Since I haven't bought the credits, it fell back to the default search feature.
And these are just examples of how powerful this apify SMTP server can be.
Of course, to set this up yourself, I will include the complete configuration schema in the resources
section of this lesson, so you can import it directly into your cloud desktop app configuration file,
and have some of the scrapers offered by Apify SMTP server ready to use.
These tools show the potential because instead of juggling multiple scraping tools and API keys, you
get intelligent data extraction that adapts to your specific needs.
So you can combine multiple scrapers in a single request.
And the LLM will act like an AI agent that intelligently decides which tools to use and in what order.
Plus, you are not limited to just data extraction.
You can add analysis, visualization, and even generate insights in one request.
I hope this was helpful.
Thanks for watching and see you in the next one.