Since the start of Bitcoin in 2009, the popularity and prevalence of cryptocurrencies has exploded, resulting in a net worth of over $1 trillion that continues to grow. Cryptocurrency—held in virtual wallets—is obtained by users who purchase coins on a cryptocurrency exchange, receive coins as payment from someone else, or “mine” coins virtually themselves. Mining is the process by which cryptocurrency providers “mint” new coins and verify transactions on existing coins already in circulation. As noted by Coinbase, “[mining] involves vast, decentralized networks of computers around the world that verify and secure blockchains—the virtual ledgers that document cryptocurrency transactions. In return for contributing their processing power, computers on the network are rewarded with new coins. It’s a virtuous circle: the miners maintain and secure the blockchain, the blockchain awards the coins, the coins provide an incentive for the miners to maintain the blockchain.” 

As with any new and relatively unregulated technology, cryptocurrency—and the process of mining it—has provided new ways malicious actors can manipulate and exploit vulnerabilities in adjacent tools and technologies for profit. With this in mind, the SafeBreach Labs research team set out to consider how attackers might be able to leverage the capabilities of the cloud to cheaply—and secretly—mine for cryptocurrency. 

Our research led us to develop the first cloud-based cryptocurrency miner utilizing Microsoft Azure’s Automation Service. We discovered three unique methods to execute the miner. Two of the methods enabled us to execute the miner within our own environment utilizing the computational resources of Microsoft servers for free. The third method allowed us to execute the miner in a victim’s environment—all while remaining completely undetected. While this research is significant because of its potential impact on cryptocurrency mining, we also believe it has serious implications for other areas, as the techniques could be used to achieve any task that requires code execution on Azure. 

Below we will share the details behind our research, beginning first with a high-level overview about how and why attackers would want to develop this type of under-the-radar crypto-mining campaign. We will then explain how Azure Automation Service works and the research process that led us to successfully exploit the service within our own environment at no cost, but also within a victim’s environment completely undetected. Finally, we will highlight the vendor response and identify how SafeBreach is sharing this information with the broader security community to help organizations protect themselves.

The Ultimate Crypto Miner 

While the concept of earning effortless income from crypto mining seems appealing, there are relatively significant limitations and challenges. Most notably, the high computational demands and energy usage associated with cryptocurrency mining have great financial costs, ultimately making the effort unprofitable for most individuals.

Therefore, some attackers create and manage crypto mining campaigns to exploit computational resources of compromised systems. To maintain these campaigns, attackers need to continuously monitor the compromised systems and perform actions to remain undetected. This costs time and of course creates the risk of being exposed to the criminal legal system.

To understand what the ultimate crypto miner would look like, let’s understand the pros and cons of each mining approach:

Approach	Pros	Cons
Mining in the attacker’s environment	Complete control over the infrastructure and mining efficiency	High maintenanceLimited resourcesHigh cost Limited profitability
Mining in the victim’s environment	No maintenance requiredVictim is charged for energy usagePotential of unlimited resources until detected	Requires access to a victim environmentRequires additional effort to evade security controlsIncludes risk of detection and resulting legal repercussions

The ultimate crypto miner would provide unlimited access to victim resources, while being maintenance-free, cost-free, and fully undetectable. Where might that be possible? In the cloud.

Azure Automation Service

Azure Automation Service is designed to automate cloud-management tasks (execute code) without building and maintaining the infrastructure. That means we only need to supply the scripts to execute, and the service does everything for us in terms of resource management. The main components of the service are: 

• Automation Account: The main resource of the service that contains all automation artifacts, such as Runbooks (scripts).
• Runbook: The primary resource within an Automation Account. It contains a script (Python or Powershell) to execute. These can be executed manually, scheduled, or triggered by events.
• Job: Every time we execute a Runbook, a new job is created. The Job resource represents the current execution and contains information about the execution such as logs, output (stdout), errors (stderr), exceptions, and execution state (e.g., running, failed, completed). Every job execution is limited to 3 hours.

A breakdown of the Job lifecycle can be seen below.

Now that we understand the basic role of the Automation Service and its features, let’s see how we can build the ultimate crypto miner with the help of this service.

Sanity Check Goes Wrong

According to Microsoft, the service charges for the job runtime only. We decided to do a sanity check to make sure we understood how the pricing actually worked. For a reliable test, we started with a clean environment.

First, we created a Python script to infinitely perform hashing calculations to simulate the central processing unit (CPU) consumption of a real miner. We chose not to execute a real miner in Azure because the license does not permit it, and our goal was to prove that it was possible without actually operating against an Azure license.

Then, we created an Automation Account and a Python 3.10 runbook that contained our script. We could execute the runbook 30 times to reach a run time of 10,800 minutes. (30 executions * 60 minutes * 3 hours). According to the Azure pricing calculator, that should cost approximately $21.60. We checked the final billing after a month to ensure the results were as we expected. To our surprise, we were not charged for a single penny. 

There appeared to be a bug in the billing calculation for Python 3.10 runbooks that allowed us to execute an infinite number of jobs totally free of charge. We reported this issue to Microsoft, and they deployed the fix to production within about a week. Our mission was complete: we found a way to execute our crypto miner maintenance-free, totally free of charge, and on unlimited resources. 

However,  this mining approach was performed on our environment. Next, we wanted to implement a miner to run in a victim’s environment. This approach had two main requirements:

• The miner must be hidden
• The miner must run infinitely (remember job runtime is limited for 3 hours)

Diving into the Service

To explore the environment the jobs are executed on, we executed a runbook to create a reverse shell towards an external server. As we received the connection, we ran some simple commands to get basic information about the runtime environment.

As noted, we were running inside a container in a Linux environment, and the main process was the ‘dotnet’ process (pid 18), which also executes our runbook.

Exploring the file system, we found the Dockerfile used to build this container at “/app/Docker/py.3.10”. We could copy the Dockerfile, along with other required dependencies to our local computer, and build the docker locally to better understand its behavior.

By capturing the network traffic, we were able to see that the main process (dotnet) communicated with an endpoint whose IP address was stored in the environment variable “AutomationServiceEndpoint”. 

This was the communication flow:

At startup, the process authenticated to the endpoint using the credentials “ClientId” and “ClientKey”, also stored in the environment variables. The process then received an access token for later communication.

A summary of the requests sent from the dotnet process to the Automation endpoint is included below.

Authorizing Requests to the Automation Endpoint

After we identified how the dotnet process communicates with the Automation service endpoint, we wanted to try to send the exact requests from within a job to see how the server would respond.

First, we needed to retrieve the access token. To do so, we tried to send a POST request with a path /connect to the Automation endpoint, but this resulted in a bad response: 401 Unauthorized. This was probably because the main process had already been authenticated to the server. So how could we retrieve the access token anyway?

The access token was saved in the dotnet process memory. Very easily, we searched for the access token in the process’s memory and  we found it.

We were then able to send requests to the Automation endpoint that were authorized successfully. As a result, we were able to gain control over the job resource information and modify it. This included the job’s logs, output, errors, and most importantly, the job’s status. This status represents the job state in the life cycle described above.

Below is an example of a running job that we masqueraded as “Stopped” in the Azure Portal.

What’s even more amazing is that the job then became unstoppable both from the Azure Portal and the API. Below is a short video demonstrating what this flow looks like:

While we succeeded in masquerading the job status, it was still possible to get the job data and see what had been executed. Next, we wanted to make it hidden.

Completely Hide a Running Job

Runbooks can also be executed in “test” mode. When a runbook is executed for testing, a job is created and referred to as a “test-job”. This feature allows you to instantly test your runbook changes before sending them to production. The test-jobs run in the same environment as regular jobs, so what are the differences between them?

	Regular Job	Test Job
Maximum running jobs at a time	Infinite	1
Log for creating a job	Yes	No (Only for modifying the runbook draft)
Retrieving job data	All jobs for the last 30 days	Only the last test-job

According to Microsoft, there are two methods to retrieve information about a test-job. The first one involves using the API request below:

The second method utilizes the ‘Test-Pane’ page in the Azure Portal (view last test), which also uses the same API above.

A user can only retrieve the current test-job’s information and has no option for retrieving information from older tests. While this sounds promising, it’s important to remember that only one test can run at the same time. So, we wanted to try to bypass that mitigation using the information we found in the previous section.

We created a test-job, which modifies its job status to “Failed”, meaning the job has finished its life cycle. This is how it looks in the Azure Portal:

At that point, the service believed that no test-job was currently running and allowed us to create another test-job. But what happened to the first test-job as we created another one? It was completely gone. It was impossible to reach this job or even know it was running. Using this flow, we were able to completely hide code execution within the Azure environment. 

Now, we wanted to focus on the last limitation: the three-hour run time limitation of the job. Fortunately, the service also provided a solution for that.

Overcome the Job Time Limit using a Hybrid Worker

Azure jobs can either run on Azure sandboxes that are managed by Azure (the container we are familiar with), or on user-managed virtual machines with the Hybrid Runbook Worker extension. Jobs running on a Hybrid Worker do not have a time limit.They also communicate with the Automation Service, but the APIs are a bit different. Luckily we had access to any code used by the Hybrid Worker extension (which is written in Python) and could use it for our needs. 

The main component of the extension was the ”jrds” client located at “/var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-1.1.14/HybridWorkerAgent/automationworker/3.x/worker/jrdsclient.py” which simply handles every request we wanted to send—even modifying the job status.

Now that we were able to work around all of the initial limitations, the full crypto-miner flow would look like this:

While we now had two flows for executing our miner without getting charged, we wanted to see if we could find an additional flow that would be totally unmonitored.

Gaining Code Execution using Python Package Upload Flow

The Azure Automation Service allows custom Python packages to be uploaded that can be used later within the runbooks scripts. Every time a package is uploaded, the service installs the package in the background to validate it is installed successfully (to prevent issues in runbook execution). Then, it saves the package in the Automation Account. We wanted to see if we could use that behavior to our advantage.

We knew that code could not be directly executed on a .whl file installation, but we could assume what command was used in the background for installation. We assumed the .whl file would be installed by the pip executable already installed in the Automation Account using the command “pip install <whl_file> ”.

If our assumptions were correct, we could create a malicious package named “pip” and upload it to the Automation Account. The upload flow would replace the current pip in the Automation account. After our custom pip was saved in the Automation account, the service used it every time a package was uploaded. So, code execution is achieved using the flow shown below.

For more details, see the package upload demo below:

To be clear, we ran this inside a temporary sandboxed environment. From within this environment, we were also able to retrieve an access token representing the Automation Account assigned identity. It could be used to send requests on behalf of that identity, making it almost impossible to trace back. The default role assigned to an Automation Account is “Contributor,” which has access to all resources in the subscription. See this Microsoft article for details about Azure’s managed identities. Below is an example from within the container:

Bonus fact: code execution could be achieved using the Powershell module upload flow. We just needed to create a Powershell module file containing any code we wanted and upload it.

And now the moment of truth: the pricing test. Since this feature pricing was not documented but we could gain code runtime, we needed to manually identify how much the flow cost. For the test, we created 55 import flows simultaneously using the method we described earlier. That means approximately 10,000 minutes of runtime as every import flow is limited for 3 hours. After a month, we saw the following results:

We were charged $0, meaning we’d found another way to execute our crypto miner maintenance-free, totally free of charge, and on unlimited resources. 

We also created a tool called CloudMiner that allows anyone to execute code for free on Azure using this technique. Once you provide the Python/Powershell script you want to execute and access to your Automation Account, the tool will handle the rest for you. Below is an example of CloudMiner usage:

Vendor Response

All issues were reported to the Microsoft Security Response Center (MSRC), though no CVEs were issued. Microsoft acknowledged these reports in their Security Researcher Acknowledgements for Microsoft Online Service.

Microsoft took action to fix the Python 3.10 free runtime issue and now prevents the job from authorizing requests to the APIs described above. With that said, they decided not to prevent/charge for the upload module/package flow as that is considered “by design.”

Below is the communication we received on the Python 3.10 runbook billing issue:

Below is the communication we received on the Job status modification issue:

Below is the communication we received on the package/module import flow code execution:

Key Takeaways

Based on the findings associated with this research, we believe there are a few important takeaways: 

• Because cloud services providers manage a huge number of resources and resource types, it can be very hard to keep track of the billing for each one. As noted above, we managed to find three ways to run unlimited code to support our cloud-based crypto miner without paying a single bill. While we only tested our theory using Azure’s Cloud Automation Service, we believe additional research would uncover more opportunities like this within Azure and other cloud providers. We believe it is critical that cloud providers remain vigilant in proactively monitoring the procedures that govern resource usage within their service offerings to better prevent such abuse.
• As cloud provider customers, individual organizations must proactively monitor every single resource and every action being performed within their environment. We highly recommend that organizations educate themselves about the methods and flows malicious actors may use to create undetectable resources and proactively monitor for code execution indicative of such behavior. Many of these activities are not monitored by default by cloud providers and security controls (remember that an upload of a Python package/PowerShell module can result in code execution and in sending requests on behalf of the Automation Account identity).

For Azure specifically, we recommend organizations monitor the following logs: 

• Write an Azure Automation runbook draft
• Create or Update an Azure Automation Python 3 package
• Create or Update an Azure Automation Python 2 package
• Create or Update an Azure Automation PowerShell module

• While the example we used in our research showed the ability of attackers to facilitate malicious cryptocurrency mining activities, we believe that is just the beginning. Attackers could easily utilize these same techniques to achieve any task that requires code execution on Azure. We believe cloud services providers and individual organizations have a shared responsibility to take the proactive steps necessary to minimize the ability of attackers to utilize resources in this way. 

Conclusion

The implications of our research are significant, as many organizations use cloud services like those provided by Microsoft Azure Cloud Automation and could be exposed to malicious actors looking to utilize their computational resources for a variety of reasons. To help mitigate the potential impact of these flows, we have: 

• Responsibly disclosed our research findings to Microsoft. While they took immediate action to address the Python 3.10 runbook billing issue, the issue regarding the package/module import flow code execution remains accessible. 
• Included access to our CloudMiner project on GitHub to be utilized by organizations wishing to do their own research and develop a better understanding of the methods and flows malicious actors may use to create undetectable resources.
• Shared our research openly with the broader security community here to raise awareness about these issues.
• Added Original Attack content to the SafeBreach platform that enables our customers to validate their security controls against these flows and significantly mitigate their risk.

For more in-depth information about this research, please: 

• Contact your customer success representative if you are a current SafeBreach customer
• Schedule a one-on-one discussion with a SafeBreach expert
• Contact Kesselring PR for media inquiries 

About Our Researchers

Ariel Gamrian is a security researcher currently working with the Threat Security Research team at SafeBreach Labs. Ariel has a well-rounded background in security that includes serving as a Company Commander during his military tenure, over a year of experience in security research, and a strong foundation in software development. Ariel’s primary areas of focus include Windows internals and cloud research.