cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

This product reached the end of support date on March 31, 2021.

IO Error: Got minus one from a read call

pawandeep_sing1
Participant

Hi Team

We are receiving lots of incidents in DT related to Performance warehouse.

It seems PW connection is getting lost, thought to check with community here. What all we can do to mitigate the situation.

Pointing in this direction will be helpful.


Here is some samples.

Sample 1:

2019-10-05 02:14:55 UTC WARNING [CITestExecutionQueue] No connection to Performance Warehouse, test executions queue will not be processed until reconnection. [log message will be suppressed for 10 minutes]
2019-10-05 02:14:55 UTC INFO [CollectorPeer] Discarded db monitor event due to missing metrics, DB Agent: URDS_CRR_Oracle [log message will be suppressed for 1 minutes]
2019-10-05 02:15:03 UTC WARNING [AgentEventChunk] RECEIVED EVENT: 29 for subAgent: fb956f81 (masterAgent: 23aeaeee) that is not registered on server. Skipping events. Suppressing this warning for another 60000 ms.
2019-10-05 02:15:10 UTC INFO [JmxMetricsManager] DssTransactionCleanupTimer wakes up at 1570241710217. Size 0. Waited for lock: 0ms
2019-10-05 02:15:10 UTC INFO [JmxMetricsManager] Metrics calculation is done in 0ms. decodeCountPastFiveMinutes: 0, decodeTimeMsPastFiveMinutes: 0
2019-10-05 02:15:10 UTC INFO [AbstractRepository] [REPOSITORY-RECONNECT-THREAD] started trying to reconnect the performance warehouse ...
2019-10-05 02:15:16 UTC INFO [DelegatingDataSource] An exception of class java.sql.SQLRecoverableException was thrown when trying to get a connection from the data source. Message : IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms., code: 17002, retry counter: 3
2019-10-05 02:15:16 UTC WARNING [DeferredAnalysisCacheSink] Failed to put or update entry in AnalysisCache @ AnalysisCacheKey [systemProfileId=, analyzerType=slowDatabaseQueries, splittings={databaseAgent=UAT_DFC_IQ_ALSCOMARCHIVE_MSSQL}] [log message will be suppressed for 1 hours]: com.dynatrace.diagnostics.common.logging.ThrottlingLogger logViaJdk:156
com.dynatrace.diagnostics.repository.handlers.analysiscache.api.AnalysisCacheHandlerException: com.dynatrace.diagnostics.server.shared.exceptions.repository.RepositoryNotConnectedException: IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms., most likely because repository is not connected (UNABLE_TO_CONNECT).
at com.dynatrace.diagnostics.repository.handlers.analysiscache.AnalysisCacheConnectionExecutionCallbackAdapter.throwOccurredException(SourceFile:33)
at com.dynatrace.diagnostics.repository.handlers.analysiscache.DefaultAnalysisCacheHandler.get(SourceFile:560)
at com.dynatrace.diagnostics.repository.handlers.analysiscache.OracleAnalysisCacheHandler.get(SourceFile:19)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStoreManager.get(SourceFile:173)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStore.addOrUpdate(SourceFile:50)
at com.dynatrace.diagnostics.server.analysiscache.store.LazyDatabaseCacheStore.addOrUpdate(SourceFile:58)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCacheEntry.addOrMerge(SourceFile:64)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCache.putOrUpdate(SourceFile:159)
at com.dynatrace.diagnostics.server.analysiscache.DeferredAnalysisCacheSink$1.run(SourceFile:44)
at com.dynatrace.diagnostics.server.ScheduledThreadPoolExecutorWithExceptionListener$3.run(SourceFile:98)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.dynatrace.diagnostics.server.shared.exceptions.repository.RepositoryNotConnectedException: IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms., most likely because repository is not connected (UNABLE_TO_CONNECT).
at com.dynatrace.diagnostics.repository.exceptiontranslators.DefaultExceptionTranslator.a(SourceFile:65)
at com.dynatrace.diagnostics.repository.exceptiontranslators.DefaultExceptionTranslator.translateException(SourceFile:60)
at com.dynatrace.diagnostics.repository.statement.DefaultStatementExecutor.writeOnTransactionalConnection(SourceFile:95)
at com.dynatrace.diagnostics.repository.handlers.analysiscache.DefaultAnalysisCacheHandler.get(SourceFile:559)
... 15 more
Caused by: com.dynatrace.diagnostics.server.shared.exceptions.repository.NoConnectionException: IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms.
at com.dynatrace.diagnostics.repository.exceptiontranslators.OracleExceptionTranslator.doTranslateException(SourceFile:91)
... 18 more
2019-10-05 02:15:17 UTC INFO [DelegatingDataSource] An exception of class java.sql.SQLRecoverableException was thrown when trying to get a connection from the data source. Message : IO Error: Got minus one from a read call, connect lapse 7079 ms., Authentication lapse 0 ms., code: 17002, retry counter: 3
2019-10-05 02:15:17 UTC WARNING [AnalysisCacheEntry] Failed to age store LazyDatabaseCacheStore[key=AnalysisCacheKey [systemProfileId=, analyzerType=databaseWaitStates, splittings={databaseAgent=UAT_DFC_IQ_ALSCOMARCHIVE_MSSQL}], delegate=DatabaseCacheStore[key=AnalysisCacheKey [systemProfileId=, analyzerType=databaseWaitStates, splittings={databaseAgent=UAT_DFC_IQ_ALSCOMARCHIVE_MSSQL}],analysisCacheKeyId=-941380567]] [log message will be suppressed for 1 hours]: com.dynatrace.diagnostics.common.logging.ThrottlingLogger logViaJdk:156
com.dynatrace.diagnostics.repository.handlers.analysiscache.api.AnalysisCacheHandlerException: No connection to the PWH. State is UNABLE_TO_CONNECT
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStoreManager.d(SourceFile:114)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStoreManager.mergeData(SourceFile:233)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStore.mergeChunks(SourceFile:123)
at com.dynatrace.diagnostics.server.analysiscache.store.LazyDatabaseCacheStore.mergeChunks(SourceFile:102)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCacheEntry.age(SourceFile:353)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCache.age(SourceFile:478)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCacheFacade$4.run(SourceFile:346)
at com.dynatrace.diagnostics.server.ScheduledThreadPoolExecutorWithExceptionListener$2.run(SourceFile:82)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


1 REPLY 1

sebastian_kryst
DynaMight Leader
DynaMight Leader

There can be plenty of reasons, network issue, problem with resources or connection pools on DB etc. Did you try some investigation in this area?

Sebastian