Upgrading IOS XR is pretty simple – add packages, activate them, commit new version and upgrade FPD on line cards if necessary. It seems so not all components were upgraded during this process. Everything went smooth but one component were left on every line card – rommonA

When command upgrade hw-module fpd all location all is executed firmware an all line cards should be upgraded but according to following output something was missed. Using force option for FPD upgrade process is not upgrading rommonA either.

RP/0/RP0/CPU0:CRS(admin)#show hw-module fpd location all 
Wed Dec 22 11:37:17.103 UTC

===================================== ==========================================
                                      Existing Field Programmable Devices
                                      ==========================================
                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/0/SP       FP40                       0.3   lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/0/CPU0     4-10GE                     0.80  lc   fpga1   0      10.00     No 
                                              lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/RP0/CPU0   RP                         0.10  lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/RP1/CPU0   RP                         0.10  lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/SM0/SP     Fabric QQS123              0.2   lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/SM1/SP     Fabric QQS123              0.2   lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/SM2/SP     Fabric QQS123              0.2   lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/SM3/SP     Fabric QQS123              0.2   lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
NOTES:
1.  One or more FPD needs an upgrade or a downgrade.  This can be accomplished
    using the "admin upgrade hw-module fpd" CLI.

Also during bootup process you can see in logs that system recommends upgrading rommonA to version 2.01.

SP/0/SM0/SP:Dec 22 10:30:31.392 : upgrade_daemon[144]: %PLATFORM-UPGRADE_FPD-4-ROMMONA_DOWN_REV : rommonA instance 0 is down-rev (V1.54), 
upgrade to (V2.01). Please consult TAC or Firmware Upgrade documentation before proceeding to upgrade rommonA. 

Ok, what is this rommonA? Every line cards have two copies of ROM Monitor. ROMMON A boots up first and if it detects presence of ROMMON B it checks its compatibility and integrity and then passes control to ROMMON B. If ROMMON B fail ROMMON A takes control.

Upgrade of rommonA have to be done manually for each card, and card have to be restarted. To do that, we have to specify that rommonA is going to be upgraded on particular card. This option won’t show in help when you press “?”.

RP/0/RP0/CPU0:CRS(admin)#upgrade hw-module fpd rommonA location 0/RP1/CPU0 
Wed Dec 22 11:40:09.796 UTC

***** UPGRADE WARNING MESSAGE: *****
  *  This upgrade operation has a maximum timout of 90 minutes.  *
  *  If you are executing the cmd for one specific location and  *
  *  card in that location reloads or goes down for some reason  *
  *  you can press CTRL-C to get back the RP's prompt.           *
  *  If you are executing the cmd for _all_ locations and a node *
  *  reloads or is down please allow other nodes to finish the   *
  *  upgrade process before pressing CTRL-C.                     *

% RELOAD REMINDER:
  - The upgrade operation of the target module will not interrupt its normal
    operation. However, for the changes to take effect, the target module
    will need to be manually reloaded after the upgrade operation. This can
    be accomplished with the use of "hw-module  reload" command.
  - If automatic reload operation is desired after the upgrade, please use
    the "reload" option at the end of the upgrade command.
  - The output of "show hw-module fpd location" command will not display
    correct version information after the upgrade if the target module is
    not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]


FPD upgrade in progress on some hardware, reload/configuration change 
on those is not recommended as it might cause HW programming failure 
and result in RMA of the hardware.


Starting the upgrade/download of following FPD:
=========== ==== ======= ======= =========== =========
                                   Current    Upg/Dng
Location    Type Subtype Upg/Dng   Version    Version
=========== ==== ======= ======= =========== =========
0/RP1/CPU0  lc   rommonA upg         1.54        2.01   
------------------------------------------------------
RP/0/RP1/CPU0:Dec 22 11:40:33.211 : upgrade_daemon[363]: Verifying /net/node0_RP0_CPU0/disk0:/hfr-fpd-4.0.1/0x13/fpd/ucode/rommon-hfr-ppc7455-asmp-A.bin: 
RP/0/RP1/CPU0:Dec 22 11:40:34.333 : upgrade_daemon[363]: Passed.
RP/0/RP1/CPU0:Dec 22 11:40:41.024 : upgrade_daemon[363]: Verifying ROMMON A: 
RP/0/RP1/CPU0:Dec 22 11:40:41.085 : upgrade_daemon[363]: Passed.
Successfully upgraded   rommonA for                   RP on location  0/RP1/CPU0 from  1.54 to  2.01


FPD upgrade has ended.

RP/0/RP0/CPU0:CRS#hw-module location 0/RP1/CPU0 reload 
Wed Dec 22 11:41:05.320 UTC
hw-module location 0/RP1/CPU0 reload will be deprecated.
Recommendation: Use hw-module location 0/RP1/* reload.
WARNING: This will take the requested node out of service.
Do you wish to continue?[confirm(y/n)]y
RP/0/RP0/CPU0:Dec 22 11:41:08.041 : redcon[309]: %HA-REDCON-1-STANDBY_NOT_READY : standby card is NOT ready 
RP/0/RP1/CPU0:Dec 22 11:41:08.075 : redcon[309]: %HA-REDCON-1-STANDBY_NOT_READY : standby card is NOT ready 

After line cards, in this case standby supervisor, boots up we can see that rommonA firmware has been upgraded.

RP/0/RP0/CPU0:CRS#show hw-module fpd location all 
Wed Dec 22 12:00:46.941 UTC

===================================== ==========================================
                                      Existing Field Programmable Devices
                                      ==========================================
                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/0/CPU0     4-10GE                     0.80  lc   fpga1   0      10.00     No 
                                              lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/RP0/CPU0   RP                         0.10  lc   rommonA 0       1.54     Yes
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
0/RP1/CPU0   RP                         0.10  lc   rommonA 0       2.01     No 
                                              lc   rommon  0       2.01     No 
--------------------------------------------------------------------------------
NOTES:
1.  One or more FPD needs an upgrade or a downgrade.  This can be accomplished
    using the "admin upgrade hw-module fpd" CLI.