Short answer
The Quran tells us Ibrahim and his son Ismail raised the foundations of the Kaaba, and that it was the first House of worship set up for mankind. Whether Adam or the angels built it before them is reported in narrations scholars do not consider authentic. The Quraysh later rebuilt it, and it has been rebuilt since.
There is no fiqh ruling here. Like all Sunni scholars, Hanafi scholars affirm what the Quran states: Ibrahim and Ismail raised the House on a site Allah showed them, and it is the first House of worship on earth.
The four schools agree on this.
Explanation
The Quran says Ibrahim was "raising the foundations of the House, and with him Ismail", praying: "Our Lord, accept this from us." It also says Allah "designated for Ibrahim the site of the House". Abu Dharr asked the Prophet ﷺ which mosque was set up on earth first, and he said: al-Masjid al-Haram, then al-Aqsa, forty years later.
- Ibrahim and Ismail: certain, from the Quran. In Bukhari's long report, Ismail brought the stones and Ibrahim built, standing on a stone as the walls rose: the Maqam Ibrahim.
- Before them: some reports say angels or Adam built it first, and that Ibrahim rebuilt on old foundations. Many scholars, including Ibn Kathir, said nothing authentic from the Prophet confirms that Adam built it. The words "raising the foundations" and "designated the site" suggest the place was known before Ibrahim.
- The Quraysh: rebuilt it when the Prophet was a young man, and he placed the Black Stone. Short of funds, they left out part of the original base, the Hijr (see Hijr Ismail).
- Later: rebuilt by Abdullah ibn al-Zubayr, then restored after that, and repaired many times since.
For more history, see Kaaba facts and what Muslims believe about the Kaaba.
The evidence
Quran, Surah Al-Baqara 2:127
وَإِذْ يَرْفَعُ إِبْرَٰهِۦمُ ٱلْقَوَاعِدَ مِنَ ٱلْبَيْتِ وَإِسْمَٰعِيلُ رَبَّنَا تَقَبَّلْ مِنَّآ ۖ إِنَّكَ أَنتَ ٱلسَّمِيعُ ٱلْعَلِيمُ ﴿١٢٧﴾
And [mention] when Abraham was raising the foundations of the House and [with him] Ishmael, [saying], "Our Lord, accept [this] from us. Indeed You are the Hearing, the Knowing.
Ibrahim and Ismail built the Kaaba.
Quran, Surah Al-Hajj 22:26
وَإِذْ بَوَّأْنَا لِإِبْرَٰهِيمَ مَكَانَ ٱلْبَيْتِ أَن لَّا تُشْرِكْ بِى شَيْـًۭٔا وَطَهِّرْ بَيْتِىَ لِلطَّآئِفِينَ وَٱلْقَآئِمِينَ وَٱلرُّكَّعِ ٱلسُّجُودِ ﴿٢٦﴾
And [mention, O Muhammad], when We designated for Abraham the site of the House, [saying], "Do not associate anything with Me and purify My House for those who perform Tawaf and those who stand [in prayer] and those who bow and prostrate.
The site was shown to Ibrahim.
Quran, Surah Aal-i-Imraan 3:96
إِنَّ أَوَّلَ بَيْتٍۢ وُضِعَ لِلنَّاسِ لَلَّذِى بِبَكَّةَ مُبَارَكًۭا وَهُدًۭى لِّلْعَٰلَمِينَ ﴿٩٦﴾
Indeed, the first House [of worship] established for mankind was that at Makkah - blessed and a guidance for the worlds.
The Kaaba is the first House of worship.
Sahih al-Bukhari 3366
Sahih... أي مسجد وضع في الأرض أول قال المسجد الحرام ...
"Which mosque was set up on earth first? He said: al-Masjid al-Haram."
The Kaaba's mosque was the first on earth.
Sahih al-Bukhari 3364
Sahih... فجعل إسماعيل يأتي بالحجارة وإبراهيم يبني ...
"Ismail began bringing the stones while Ibrahim built."
How Ibrahim and Ismail built the House.
Checked by our team. For a more accurate answer to your own situation, consult a scholar. Quran translation by Sahih International. Hadith Arabic from the standard editions of the six books and the Muwatta, with gradings as recorded there; the English is our own rendering. Spotted a mistake? Tell us.