Versions Compared

Old Version 17

changes.mady.by.user Cintia Del Rio

Saved on

compared with

New Version Current

changes.mady.by.user Cintia Del Rio

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Note

This page hasn't been updated as much as it should :/

 

 

 

ServiceDate/time UTCResolutionCauseComments
Confluence   https://talk.openmrs.org/t/openmrs-wiki-down/8062
Sonar ~11amRestart service(question)

Sonar is not on status page

Builds failed due to outage

Jira  22:44restart serviceOut of memory
Expand

2016-09-22 21:10:15,849 ajp-bio-8041-exec-560 WARN anonymous 1266x78860x6 - 50.23.94.74 /secure/Dashboard.jspa [renderer.internal.local.LocalGadgetSpecFactory$Parser] Could not retrieve gadget spec https://issues.openmrs.org/rest/gadgets/1.0/g/com.atlassian.jira.gadgets:filter-results-gadget/gadgets/filter-results-gadget.xml from plugin-provided gadget spec store: java.lang.OutOfMemoryError: Java heap space
2016-09-22 21:10:15,925 ajp-bio-8041-exec-547 WARN anonymous 1266x78862x8 - 188.138.124.110 /secure/Dashboard.jspa [renderer.internal.local.LocalGadgetSpecFactory$Parser] Could not retrieve gadget spec https://issues.openmrs.org/rest/gadgets/1.0/g/com.atlassian.jira.gadgets:filter-results-gadget/gadgets/filter-results-gadget.xml from plugin-provided gadget spec store: java.lang.OutOfMemoryError: Java heap space
Exception in thread "ajp-bio-8041-Acceptor-0" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ajp-bio-8041-exec-433" java.lang.OutOfMemoryError: Java heap space

Confluence 23:26Restart ServiceAdded new profiling flag 
Jira 23:38Restart ServiceCrashed 
Bamboo ~00:00Kill old service and start againCrashed?

For some reason, Bamboo server was printing some very weird error messages to the logs:

Expand

Sep 26, 2016 9:33:20 PM com.sun.jersey.spi.container.ContainerResponse write

SEVERE: A message body writer for Java class com.atlassian.bamboo.plugins.rest.model.server.RestServerStatusInfo, and Java type class com.atlassian.bamboo.plugins.rest.model.server.RestServerStatusInfo, and MIME media type application/xml was not found

Sep 26, 2016 9:33:20 PM com.sun.jersey.spi.container.ContainerResponse write

SEVERE: The registered message body writers compatible with the MIME media type are:

application/xml ->

  com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$App

  com.sun.jersey.core.impl.provider.entity.DocumentProvider

  com.sun.jersey.core.impl.provider.entity.SourceProvider$SourceWriter

  com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$App

  com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$App

*/* ->

  com.sun.jersey.core.impl.provider.entity.FormProvider

  com.sun.jersey.core.impl.provider.entity.StringProvider

  com.sun.jersey.core.impl.provider.entity.ByteArrayProvider

  com.sun.jersey.core.impl.provider.entity.FileProvider

  com.sun.jersey.core.impl.provider.entity.InputStreamProvider

  com.sun.jersey.core.impl.provider.entity.DataSourceProvider

  com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$General

  com.sun.jersey.core.impl.provider.entity.ReaderProvider

  com.sun.jersey.core.impl.provider.entity.DocumentProvider

  com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider

  com.sun.jersey.core.impl.provider.entity.SourceProvider$SourceWriter

  com.sun.jersey.server.impl.template.ViewableMessageBodyWriter

  com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General

  com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$General

  com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$General

  com.sun.jersey.json.impl.provider.entity.JSONWithPaddingProvider

  com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$General

  com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$General

  com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$General

  com.sun.jersey.json.impl.provider.entity.JSONListElementProvider$General

  com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy

It appears to be some sort of red herring, because this error appears on the logs every so often.

 

Also, service stop didn't work, I had to use kill -9. I forgot to reenable the agents, bringing the outage to pretty much a day.

jira restart servicecrashed 
Bamboo 

Re-authorise agents in bamboo

?
Expand

Nov 16, 2016 11:55:37 AM Remote agent 'gw107.iu.xsede.org (1)' was unresponsive and has gone offline.
Nov 16, 2016 11:56:05 AM Remote agent 'gw107.iu.xsede.org (2)' was unresponsive and has gone offline.
Nov 16, 2016 11:56:33 AM Remote agent 'gw108.iu.xsede.org (1)' was unresponsive and has gone offline.
Nov 16, 2016 11:56:37 AM Remote agent 'gw108.iu.xsede.org (2)' was unresponsive and has gone offline.
Nov 17, 2016 7:45:18 AM A remote agent is loading on gw107.iu.xsede.org (127.0.0.1).
Nov 17, 2016 7:45:54 AM A remote agent is loading on gw108.iu.xsede.org (127.0.0.1).
Nov 17, 2016 7:45:55 AM A remote agent is loading on gw108.iu.xsede.org (127.0.0.1).
Nov 17, 2016 7:46:03 AM Remote agent [gw107.iu.xsede.org] came back after a period of inactivity.
Nov 17, 2016 7:46:03 AM A remote agent is loading on gw107.iu.xsede.org (127.0.0.1).
Nov 17, 2016 7:46:18 AM Remote agent [gw108.iu.xsede.org (3) (2)] came back after a period of inactivity.
Nov 17, 2016 7:46:46 AM Remote agent [gw108.iu.xsede.org (4) (2)] came back after a period of inactivity.
Nov 17, 2016 7:46:56 AM Remote agent [gw107.iu.xsede.org (2)] came back after a period of inactivity.