Experiencing disruptions
Core   (?) Testing Farm Core Operational
Public Ranch   (?) Testing Farm Public Ranch Operational
Red Hat Ranch   (?) Testing Farm Red Hat Ranch Disrupted
BaseOS CI   (?) BaseOS CI system Operational
Cyborg Development Cluster   (?) Red Hat Cyborg Development Cluster Operational
Cyborg Loki   (?) Red Hat Cyborg Loki Deployment Operational
Fedora CI / installability   (?) Affecting Fedora CI installability test Operational

Incident history


2024 (10)

March 28, 2024 at 12:00 AM UTC

Red Hat Ranch Degraded

▲ This issue is not resolved yet
March 28, 2024 at 12:00 AM UTC

Public Ranch Outage - Expired images

Resolved in under a minute
March 19, 2024 at 12:00 AM UTC

Public Ranch Outage - Artemis requests in error state

Resolved in under a minute
March 12, 2024 at 3:00 PM UTC

Long queues in Public and Red Hat ranch

Resolved after 29h 0m of downtime
March 8, 2024 at 8:00 AM UTC

Public Ranch artifact storage ran out of disk space

Resolved after 5h 0m of downtime
March 7, 2024 at 10:00 AM UTC

Public Ranch large queue - expect delays

Resolved after 6h 0m of downtime
March 5, 2024 at 3:00 PM UTC

Public Ranch large queue - expect delays

Resolved after 6h 0m of downtime
February 15, 2024 at 5:00 PM UTC

Public Ranch large queue - expect delays

Resolved in under a minute
February 2, 2024 at 8:00 AM UTC

BaseOS CI suspended for urgent jenkins controller upgrade.

Resolved after 60m of downtime
January 3, 2024 at 10:47 AM UTC

BaseOS CI suspended for urgent jenkins controller upgrade.

Resolved after 1h 43m of downtime

2023 (41)

December 10, 2023 at 12:00 AM UTC

Public Ranch Outage - Artemis requests in error state

Resolved after 35h 0m of downtime
December 2, 2023 at 12:00 AM UTC

Public Ranch Outage

Resolved after 3h 0m of downtime
November 29, 2023 at 12:00 AM UTC

Public Ranch reservations are broken

Resolved after 202h 0m of downtime
November 23, 2023 at 12:00 AM UTC

Fedora CI installability test affected by systemd/selinux issue.

Resolved after 1882h 0m of downtime
November 20, 2023 at 12:00 AM UTC

Testing Farm Public Ranch outage due to expired images.

Resolved after 10h 0m of downtime
November 19, 2023 at 6:00 AM UTC

Testing Farm Red Hat Ranch outage during the weekend

Resolved after 27h 0m of downtime
November 12, 2023 at 12:00 AM UTC

BaseOS CI limited due to outage due to BRQ network maintanance.

Resolved after 36h 0m of downtime
November 7, 2023 at 3:00 PM UTC

BaseOS CI and Testing Farm Red Hat Ranch outage due to Beaker unavailability

Resolved after 4h 0m of downtime
September 30, 2023 at 4:00 AM UTC

Expired images on Public ranch.

Resolved after 48h 0m of downtime
September 25, 2023 at 12:00 AM UTC

Testing Farm Red Hat ranch and BaseOS CI outage for RHEL9 and RHEL-8.1.0

Resolved after 16h 30m of downtime
September 21, 2023 at 12:00 AM UTC

BaseOS CI outage due to network issues

Resolved in under a minute
September 18, 2023 at 2:00 PM UTC

Console logs are disabled due to a discovered bug

Resolved in under a minute
September 17, 2023 at 8:00 AM UTC

Red Hat Ranch outage during weekend

Resolved after 8h 0m of downtime
September 15, 2023 at 8:00 AM UTC

Console log has wrong link in results viewer or not available at all.

Resolved after 274h 0m of downtime
September 11, 2023 at 8:00 AM UTC

Composes not updating on Red Hat ranch

Resolved in under a minute
September 6, 2023 at 8:00 AM UTC

Cyborg Loki is being migrated to S3 storage and better deployment

Resolved after 2788h 0m of downtime
August 21, 2023 at 8:00 AM UTC

Beaker testing blocked on Red Hat Ranch and BaseOS CI

Resolved after 4h 0m of downtime
August 7, 2023 at 12:00 AM UTC

Large queue times on Red Hat Ranch

Resolved after 8h 22m of downtime
July 27, 2023 at 9:00 AM UTC

Red Hat Ranch affected by large queue

Resolved after 3h 11m of downtime
July 19, 2023 at 8:00 AM UTC

Public ranch testing blocked

Resolved after 3h 0m of downtime
July 18, 2023 at 8:00 AM UTC

Fedora Rawhide testing blocked on dnf5 breakage

Resolved after 6h 0m of downtime
June 22, 2023 at 9:00 AM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 528h 0m of downtime
June 21, 2023 at 12:30 PM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 4h 30m of downtime
June 1, 2023 at 10:30 AM UTC

Red Hat Ranch affected by RH infra outage

Resolved after 21h 30m of downtime
May 29, 2023 at 2:00 AM UTC

Expired Fedora-Rawhide images in Public Ranch

Resolved after 5h 0m of downtime
May 5, 2023 at 2:00 AM UTC

Unexpected outage due to new release 2023-04.1

Resolved after 10h 0m of downtime
May 2, 2023 at 8:00 AM UTC

Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Resolved in under a minute
April 28, 2023 at 4:00 AM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m of downtime
April 18, 2023 at 2:00 PM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m of downtime
April 14, 2023 at 3:00 AM UTC

Public Ranch - expired images

Resolved in under a minute
April 14, 2023 at 12:00 AM UTC

CentOS Stream 9 aarch64 broken in Public Ranch

Resolved after 92h 0m of downtime
April 6, 2023 at 10:00 AM UTC

Red Hat Ranch - deploying 2023-03.1 release - large queue expected

Resolved after 4h 0m of downtime
April 3, 2023 at 8:00 AM UTC

Red Hat Ranch - provisioning outage

Resolved in under a minute
March 29, 2023 at 8:00 AM UTC

Red Hat Ranch - larger queue

Resolved after 60m of downtime
March 28, 2023 at 8:00 PM UTC

Public ranch - Artemis resource ended in 'error' state

Resolved after 13h 0m of downtime
March 16, 2023 at 8:00 AM UTC

CentOS Stream 8 broken in Public Ranch

Resolved after 99h 0m of downtime
March 10, 2023 at 8:00 AM UTC

Fedora-Rawhide broken in Red Hat Ranch

Resolved after 148h 0m of downtime
February 10, 2023 at 9:00 AM UTC

Red Hat internal infrastructure outage

Resolved after 33h 0m of downtime
January 30, 2023 at 4:00 PM UTC

BaseOS CI - large queue, expect delays.

Resolved after 338h 0m of downtime
January 25, 2023 at 10:00 AM UTC

Public Ranch - large queue, expect delays.

Resolved after 7h 0m of downtime
January 24, 2023 at 3:00 PM UTC

Public Ranch outage - requests stuck in queued

Resolved after 19h 0m of downtime

2022 (19)

December 6, 2022 at 8:00 AM UTC

BaseOS CI Outage

Resolved after 9h 0m of downtime
November 30, 2022 at 9:00 AM UTC

Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Resolved after 7h 30m of downtime
November 29, 2022 at 8:00 AM UTC

Red Hat Ranch outage - artifacts storage empty

Resolved after 2h 30m of downtime
November 22, 2022 at 2:00 PM UTC

Public Ranch Outage - artifacts storage out of inodes

Resolved after 6h 0m of downtime
October 31, 2022 at 8:00 AM UTC

BaseOS CI Outage - UMB message sending

Resolved after 4h 0m of downtime
October 10, 2022 at 8:00 AM UTC

Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Resolved after 30h 20m of downtime
September 22, 2022 at 6:00 AM UTC

Public Ranch outage - Copr API v2 is deprecated

Resolved after 18h 30m of downtime
August 12, 2022 at 6:00 PM UTC

BaseOS CI Outage

Resolved after 67h 0m of downtime
August 11, 2022 at 8:00 AM UTC

CentOS Stream Koji Outage

Resolved after 2h 0m of downtime
August 1, 2022 at 11:00 AM UTC

Cyborg Development Cluster outage

Resolved after 2h 0m of downtime
June 16, 2022 at 9:00 AM UTC

CentOS Stream 9 repositories broken in CI

Resolved after 6h 0m of downtime
June 2, 2022 at 8:00 AM UTC

Cyborg Development Cluster credentials outage

Resolved after 7h 0m of downtime
March 25, 2022 at 12:00 PM UTC

Covscan testing blocked on BaseOS CI

Resolved after 72h 0m of downtime
February 8, 2022 at 12:00 PM UTC

rpminspect fails with "internal libpod error"

Resolved after 1441h 0m of downtime
February 5, 2022 at 8:00 PM UTC

Fedora CI outage - AWS images are gone

Resolved after 49h 25m of downtime
January 31, 2022 at 10:00 PM UTC

Broken rpmdeplint and rpminspect in Fedora CI

Resolved after 40h 30m of downtime
January 27, 2022 at 10:30 AM UTC

Large queue times on Red Hat Ranch

Resolved after 698h 22m of downtime
January 3, 2022 at 5:00 PM UTC

Large queue times on Public Ranch

Resolved after 27h 0m of downtime
January 3, 2022 at 12:17 AM UTC

Known issues  ℹ

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

2021 (3)

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m of downtime
December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m of downtime
December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m of downtime