In Azure, Infrastructure as Code is the process of provisioning and configuring resources through machine-readable definition files, rather than physical hardware config or interactive tools.
You may be familiar with Terraform and the way in which it integrates with GitHub. Terraform is a common platform used for IaC and it is popular with DevOps teams because IaC is an enabler of CI/CD practices. Azure Bicep aims to simplify deploying resources in Azure by providing a simpler, admin-friendly interaction with ARM.

If we boil it down, Azure in simple terms is all about capacity. You can purchase various services from Azure but ultimately, you're buying compute, storage and network capacity, supporting the software that glues it all together. When combined, we get 'resources' like virtual machines, virtual networks and databases. Everything you do in the Azure portal or shell is interacting with the cloud platform in JSON (language) via ARM (orchestration).
The key advantage of Bicep (another kind of language) over other IaC input tooling, besides being native, is that it has a deeper relationship with ARM. For example, if you deploy a template that won't work, Bicep will tell you before it runs "This won't work because of 'X', please do 'Y'". In contrast, Terraform relies on a state file and will run your script right up until the point of failure before telling you it's not going to work, so it's by nature, more reactive than the proactive capabilities of the baked-in Bicep service.
Getting Started with Azure Bicep
Install the Bicep AZ-CLI file (Mac, Linux, Windows) by downloading it from GitHub.
If you already have AZ-CLI 2.20.0+ installed, it may already include the Bicep CLI commands.
If you have an older version of a CLI installed and it's further up in the system path on your local machine, it will get used by PowerShell first. Check System Properties, Edit Environment Variable and delete the older version.
PowerShell Module 5.6.0+ does actually link to the Azure Bicep CLI installation in the path, making the previous point event more important.
Use 'Bicep-V' to check your installed version and 'Az Version' to check your shell version.
If using Visual Studio (writing in VS-Code) there is a Bicep extension/Plug-in that does not require the CLI or PowerShell module, and it updates itself automatically.
Once installed, I highly recommend you run through the Microsoft Azure Bicep tutorial on getting started.
Useful Links
Bicep language for deploying Azure resources - Azure Resource Manager | Microsoft Docs
Learn modules for Bicep - Azure Resource Manager | Microsoft Docs