MEMX Status - Matching Engine Issue

Matching Engine Issue - Resolved

Incident Report for MEMX

Postmortem

At 09:32:09 ET on January 11, 2023, one of MEMX’s 52 matching engines, crashed due to a software error. A total of 227 symbols assigned to the matching engine were impacted. Due to the nature of the software error, the back-up version of the matching engine also crashed.

At 09:45:50 ET, MEMX disabled the impacted matching engine resulting in electronic outs for resting orders associated with the failed matching engine and halting affected symbols. At this time an additional 438 symbols were halted due to a secondary issue in the matching engine recovery process. Out of an abundance of caution, MEMX decided to cease the recovery process, leaving a total of 665 symbols halted for the remainder of the trading day.

Root cause analysis of the matching engine crash uncovered latent software error. The root cause analysis of the secondary issue in the matching engine recovery process uncovered a separate latent software error.

Corrective software patches have been deployed for trading on January 12, 2023, to resolve both issues. MEMX systems will be operating normally for tomorrow’s trading session.

MEMX Market Operations | mops@memx.com | 1-833-415-6677

Posted Jan 11, 2023 - 21:15 EST

Resolved

MEMX has resolved the matching engine issue. A post mortem summary will be provided.

MEMX Market Operations | mops@memx.com | 1-833-415-6677

Posted Jan 11, 2023 - 20:44 EST

Update

MEMX has decided to leave the previously messaged 665 symbols halted for the remainder of the trading day. MEMX has identified the software error that caused this morning’s issue and will provide further detail in a post mortem summary this evening.

MEMX Market Operations | mops@memx.com | 1-833-415-6677

Posted Jan 11, 2023 - 12:11 EST

Monitoring

Due to a subsequent issue with the recovery process for the matching engine failure, the following symbols will remain halted on MEMX until further notice. All other systems are operating normally.

BN
APCX
IQDE
FYLD
PRTK
KPLTW
HYFM
CTO
VEEV
ZXYZ.A
SPXT
JEMA
BEPH
RC-E
GECCN
ICCC
MGEE
AACIU
LGVCU
BLDG
DQ
FEP
RCACU
CRMD
SKOR
BWAQW
KALU
AMGN
BSFC
AZTD
OMI
SG
INNO
HZON+
AHT-I
SDGS
SEDA=
BHR
MUFG
ASPS
WIP
CI
YNDX
BITF
HCNE
ADEA
PHM
BHVN
OCCIO
INC
SONDW
OSK
EFHTR
NHTC
FNGG
MITQ
ESPO
BDTX
SCCD
IVES
NJAN
PRPO
ECXWW
SSFI
CLST
PAY
NGE
ICU
ENTFW
USIG
LTBR
SWAGW
WINVW
SEVN
KIQ
RF-B
LSTA
NVCN
SQLV
FSTX
PUCK
RJF-A
ORFN
CPTN
HUDA
RYJ
BRDS
PSA-H
IR
EVX
UTSL
AAU
SGOV
CUBB
CPZ
EFHT
VZIO
IGN
IEMG
FLMX
ASH
FCTR
PSNY
NUO
PAAS
MTAL+
PSA-S
JBLU
EXG
ZVIA
FBT
SPSB
ADPT
VMO
IDHQ
MEDS
IMAQU
CZNC
AFG
GT
GTBP
FRONU
CBZ
PACI
RVSNW
MTVR
FHLT
BRP
FEMB
MOTE
MCACR
ACABU
EVLV
EAR
PYXS
NVSAU
PETWW
AOD
HIFS
AGM-F
MEME
PHYT
OVF
BTWNU
BRSHW
ITQ
ALG
MODL
AAMC
FSCO
KAIIU
VHAQ
CMA
AMRS
SPNT-B
BHFAM
ASTS
IONQ
COMS
JAQC
MPRAW
SOXX
VALN
AMWD
PLAB
PLTNW
THYF
YPS
CAG
AAME
ABM
SDACU
ADPV
NXE
SBR
AMSF
PPHPW
CLBR+
SCU
SPHD
ARR
BYTE
CLM
NUSI
OAIM
DRMAW
OCFCP
NYCB-U
ECC-D
RNEW
ZT
RWAYZ
RUTH
CRDO
GOEX
QTUM
SEIM
NBCM
FUTY
GGB
FRSX
KMB
CFCV
SNPX
PRPH
CHUY
TALKW
VDC
LFLY
CUT
IMCB
AIVI
DIM
ENZL
MIY
OPAD+
WTI
AAA
IBCE
VBLT
MDNA
EDI
EAD
NTCO
CNO
NTST
AGIH
PGRWW
PDI
OPA
NEWTZ
ITAQU
FRST
USNA
ISPOW
TEQI
AMNA
BAC-Q
MGV
PKI
IRDM
IMRX
BMAQW
VPC
TCOA+
XOMAO
UMH-D
TREE
GAU
HMPT
GOLF
HFFG
MAIN
LKCO
LIXT
NOVVU
APMI
ACXP
LGAC
GRCL
FXN
FE
AYI
JFR
ALOR
SPXU
WNDY
XRTX
CEF
GORO
BNOV
EQWL
SMTC
FVRR
GAQ+
PRPC=
BWAQ
WNW
RQI
VIS
STRR
DCP
PSDN
TNP-E
FRME
TCI
FALN
ACAHU
WMC
VOR
VALE
HDGE
FAIL
DLCAW
DKL
VTR
TCBIO
QDYN
GSP
COMM
PB
NRDS
DAX
GRF
RULE
TETC
SPPP
PL+
DATS
CATO
XNCR
SKIL
SOXS
RZB
FTXN
SURF
DSTL
PTNR
USMF
LMST
OVM
CGTX
CBFV
NBDS
SAVA
MXC
SDG
MLACU
CDXC
LHX
RWX
TIPX
CCZ
RBT
OFED
PTH
PVH
BDRY
QCRH
PCK
AREB
NIM
NM-H
ORTX
ALK
ACP
MSFD
NEOG
MPRAU
ABCB
MNSO
BALT
VSACW
KULR
CTIC
FTS
BZH
CFG-D
KROS
MSVB
ADERW
TMUS
MSGE
BWCAW
BOWL
IVCBW
BLW
BFS-D
UGI
STXK
KIE
JCTR
HCMA
GBUY
EVAX
VVI
FRBA
TYD
NCZ-A
RLAY
ICNC
IVW
GSHD
ATAQ
UZD
DTW
VNLA
IHI
GASS
KKR-C
RSX
EXFY
MIRM
GTLB
ELSE
BKSE
AGM-E
TRTN-A
ARKQ
ACQRW
ADMP
AAT
XRMI
VECO
IAC
AAOI
CVII+
SPGP
DMDV
RCLF
HERAW
ERC
TWO-A
STOK
FHN-B
ANGHW
SOHU
GETY
FSMO
VSH
FINM
STRA
CWT
GNW
EMLDU
EMGF
DLY
TCRX
RCFA
PBYI
SHOO
EWK
CCOI
VERO
RTM
INDIW
SBET
RDIB
PCG-B
DAC
PBE
RYE
OCIO
NWBI
TWO-C
NVT
ACHR
ADP
AEACU
AEYE
AFGE
AFMC
AKLI
ALEC
AMC
AMJ
AOR
APGB=
ARRWU
ATHA
AVPT
BAH
BBC
BBEU
BBSA
BEP
BKKT+
BLLD
BOTJ
BSBR
BTF
BUFD
BWNB
CAAP
CAJ
CDR-B
CHH
CKPT
CLBR=
CLX
CMRE-C
CPB
CQP
CSLM
CTBB
CTEX
CVE+
CWS
CXSE
CYB
CYTHW
DCTH
DFSU
DHF
DMYY=
DSJA
DTSS
DUNEU
DURA
ECC
EDSA
EGBN
EMN
EMP
EMXF
ESBA
ETI-
EZU
FDLO
FHN-C
FLFVR
FMNY
FMY
FRA
FRC
FTCS
FTI
FTQI
GDXJ
GECCO
GJS
GJT
GNE
GNR
GRN
GS-J
GTEC
GTX
GWH+
HCOM
HCSG
HOFT
HPK
HSRT
HSTO
HYLD
IBER
ICLR
ICUCW
IEZ
IJJ
INLX
IPO
IQDG
IRTC
IWFG
IWFH
IXG
JJA
JSD
JUPWW
KB
KMPB
KN
KNSW
KOD
KPOP
KRNLW
KVHI
LATG
LAZ
LFMDP
LGST
LOCL
LOOP
LUCD
LXEH
MAAX
MBAC+
MBNE
MBSD
MDWT
MERC
MG
MILN
MLAB
MMD
MNK
MODD
MORN
MRBK
MTN
NBSTU
NEXT
NHI
NIB
NMS
NXN
NXTC
NYMTL
OACP
OALC
OCTW
OGE
OKTA
OPCH
OSCV
OXSQZ
PAVM
PAYC
PBLA
PFLD
PFXNZ
PGTI
PICK
PILL
QLGN
QQC
QVAL
QVMM
RAD
RBCP
REUN
RFV
RGF
RHCB
RICK
RILYO
RIOT
ROSC
RRR
RWJ
SAL
SCE-G
SDS
SEM
SERA
SESG
SFM
SKM
SLV
SNN
SNPS
SPE
SRC
SRET
SSO
STER
STGW
STR
SVIIU
SWAR
TELL
TOL
TRDA
TRFK
TRTN-D
TSQ
TWND
TZPSW
ULST
UPTD
URGN
VCXA
VLAT
VQS
VWE
WBIY
WOOD
XLSR
XSMO
XYLG
YMAR
ZNTL
ZYNE

Posted Jan 11, 2023 - 11:00 EST

Update

MEMX is currently in the process of recovering the impacted matching engine. Additional information to follow.

MEMX Market Operations | mops@memx.com | 1-833-415-6677

Posted Jan 11, 2023 - 10:11 EST

Identified

MEMX has identified an issue with a single matching engine. All customers should have received information regarding any open orders. the symbols impacted as listed below:

ACHR
ADP
AEACU
AEYE
AFGE
AFMC
AKLI
ALEC
AMC
AMJ
AOR
APGB=
ARRWU
ATHA
AVPT
BAH
BBC
BBEU
BBSA
BEP
BKKT+
BLLD
BOTJ
BSBR
BTF
BUFD
BWNB
CAAP
CAJ
CDR-B
CHH
CKPT
CLBR=
CLX
CMRE-C
CPB
CQP
CSLM
CTBB
CTEX
CVE+
CWS
CXSE
CYB
CYTHW
DCTH
DFSU
DHF
DMYY=
DSJA
DTSS
DUNEU
DURA
ECC
EDSA
EGBN
EMN
EMP
EMXF
ESBA
ETI-
EZU
FDLO
FHN-C
FLFVR
FLMX
FMNY
FMY
FRA
FRC
FTCS
FTI
FTQI
GDXJ
GECCO
GJS
GJT
GNE
GNR
GRN
GS-J
GTEC
GTX
GWH+
HCOM
HCSG
HOFT
HPK
HSRT
HSTO
HYFM
HYLD
IBER
ICLR
ICUCW
IEZ
IJJ
INLX
IPO
IQDG
IRTC
IWFG
IWFH
IXG
JJA
JSD
JUPWW
KB
KMPB
KN
KNSW
KOD
KPOP
KRNLW
KVHI
LATG
LAZ
LFMDP
LGST
LOCL
LOOP
LUCD
LXEH
MAAX
MBAC+
MBNE
MBSD
MDWT
MERC
MG
MILN
MLAB
MMD
MNK
MODD
MORN
MPRAU
MRBK
MTN
NBSTU
NEXT
NHI
NIB
NMS
NXN
NXTC
NYMTL
OACP
OALC
OCTW
OGE
OKTA
OPCH
OSCV
OXSQZ
PAVM
PAYC
PBLA
PFLD
PFXNZ
PGTI
PICK
PILL
QLGN
QQC
QVAL
QVMM
RAD
RBCP
REUN
RFV
RGF
RHCB
RICK
RILYO
RIOT
ROSC
RRR
RWJ
RWX
SAL
SCE-G
SDGS
SDS
SEM
SERA
SESG
SFM
SKM
SLV
SNN
SNPS
SPE
SPGP
SRC
SRET
SSO
STER
STGW
STR
SVIIU
SWAR
TELL
TOL
TRDA
TRFK
TRTN-D
TSQ
TWND
TZPSW
ULST
UPTD
URGN
USNA
VCXA
VLAT
VQS
VWE
VZIO
WBIY
WOOD
XLSR
XSMO
XYLG
YMAR
ZNTL
ZYNE

Posted Jan 11, 2023 - 09:53 EST

Investigating

MEMX is currently investigating an issue in multiple symbols. Additional information to follow.

MEMX Market Operations | mops@memx.com | 1-833-415-6677

Posted Jan 11, 2023 - 09:38 EST

This incident affected: MEMX U.S. Equities Exchange.