Source or destination buffers passed to pagecache functions may belong
to kernel memory (e.g. when the caller is packagefs). Because of that
we should tell vm_memcpy_{from, to}_physical() truth, not assume that all
buffers are in user memory. That's important because user memory page fault
handlers cannot be nested and these functions may be used while handling
a page fault.
With high probability fixes#11246.
Signed-off-by: Paweł Dziepak <[email protected]>
This patch adds user_access() which can be used to gracefully handle
page faults that may happen when accessing user memory. It is used
by arch_cpu_user{memcpy, memset, strlcpy}() to allow using optimized
functions from the standard library.
Currently only x64 uses this, but nothing really is arch specific here.
Signed-off-by: Paweł Dziepak <[email protected]>
This is a major rework of how Haiku implements memset() and memcpy() on
x64. These functions are now removed from the commpage and reimplemented
in C++ using sse2 where it proved to be beneficial. That required some
serious changes in handling fpu state. Now, full fpu state is saved
at each interrupt, but not on task switch.
Some numbers: results of building targets: kernel libroot.so runtime_loader
and HaikuDepot on Intel i7 4770 with 16GB of memory.
real
user
sys
before:
1m54.367
7m40.617
0m58.641
1m33.922
8m12.362
1m0.852
1m32.922
8m10.509
1m1.006
1m31.947
8m12.596
1m1.353
after:
1m50.959
7m43.118
0m58.923
1m30.644
8m6.385
1m0.584
1m31.549
8m7.976
0m59.792
1m31.546
8m6.733
1m0.242
As Alex pointed out we can leak possibly sensitive data in xmm registers
when returning from the kernel. To prevent that xmm0-15 are zeroed
before sysret or iret. The cost is negligible.
Signed-off-by: Paweł Dziepak <[email protected]>
The kernel is allowed to use fpu anywhere so we must make sure that
user state is not clobbered by saving fpu state at interrupt entry.
There is no need to do that in case of system calls since all fpu
data registers are caller saved.
We do not need, though, to save the whole fpu state at task swich
(again, thanks to calling convention). Only status and control
registers are preserved. This patch actually adds xmm0-15 register
to clobber list of task swich code, but the only reason of that is
to make sure that nothing bad happens inside the function that
executes that task swich. Inspection of the generated code shows
that no xmm registers are actually saved.
Signed-off-by: Paweł Dziepak <[email protected]>
Enable SSE as a part of the "preparation of the environment to run any
C or C++ code" in the entry points of stage2 bootloader.
SSE2 is going to be used by memset() and memcpy().
Signed-off-by: Paweł Dziepak <[email protected]>
Just following the path of least resistance and adding andq $~15, %rsp
where appropriate. That should also make things harder to break
when changing the amount of stuff placed on stack before calling the
actual syscall routine.
Signed-off-by: Paweł Dziepak <[email protected]>
There is absolutely no reason for these functions to be in commpage,
they don't do anything that involves the kernel in any way.
Additionaly, this patch rewrites memset and memcpy to C++, current
implementation is quite simple (though it may perform surprisingly
well when dealing with large buffers on cpus with ermsb). Better
versions are coming soon.
Signed-off-by: Paweł Dziepak <[email protected]>
The possibility to specify custom memcpy and memset implementations
in cpu modules is currently unused and there is generally no point
in such feature.
There are only 2 x86 vendors that really matter and there isn't
very big difference in performance of the generic optmized versions
of these funcions across different models. Even if we wanted different
versions of memset and memcpy depending on the processor model or
features much better solution would be to use STT_GNU_IFUNC and save
one indirect call.
Long story short, we don't really benefit in any way from
get_optimized_functions and the feature it implements and it only adds
unnecessary complexity to the code.
Signed-off-by: Paweł Dziepak <[email protected]>
long_mmu_init() prepares initial paging structures for 64 bit kernel.
Once that function completes bootloader cannot allocate any memory
that needs to be passed to the kernel.
Signed-off-by: Paweł Dziepak <[email protected]>
Both data array and its size should be constant so that GCC can do
as much as possible at compile time and it is safe to use them in
static assertions.
This patch also changes the type of size constant to size_t which
more appropriate and uses sizeof() to determine the size of the
array.
Fixes build breakage introduced in e547662664.
Signed-off-by: Paweł Dziepak <[email protected]>
As discussed on the ML the limitation of the gap between segments
imposed by this check is completely artifical and pointless.
Signed-off-by: Paweł Dziepak <[email protected]>
cpuid is available in user mode as well and it doesn't look like there
are going to be any x86 platforms with significantly different CPUs anytime
soon.
Signed-off-by: Paweł Dziepak <[email protected]>
GCC knows whether these functions need to be implemented using syscalls
(or more clever solutions like in Linux) and calls libgcc in such case.
Signed-off-by: Paweł Dziepak <[email protected]>
The less assembler in our sources the better. These functions wouldn't
be used very much since SupportDef.h inlines them, but the symbols should
be available.
Signed-off-by: Paweł Dziepak <[email protected]>
Time to get rid of some asm code. Surprisingly, it appears that
on x86[_64] the emitted code for atomic_test_and_get() isn't as efficient
as it could be, even with -O2, but cmpxchg is so expensive that this slight
difference shouldn't matter much.
Signed-off-by: Paweł Dziepak <[email protected]>
If GCC knows what these functions are actually doing the resulting
code can be optimized better what is especially noticeable in case of
invocations of atomic_{or,and}() that ignore the result. Obviously,
everything is inlined what also improves performance.
Signed-off-by: Paweł Dziepak <[email protected]>
The way we handle paging is very wasteful and relies heavily on virtual
funcions even if there is absolutely no reason to do so. The proper
solution would be to do a major rework of paging code (including
arch-independent parts).
On x86_64 physical page mapper is very simple what makes the overhead
resulting from the desing of paging interface very expensive. This
patch attempts to make things a bit better by helping GCC with
devirtualization and allowing inlining physical page mapper impementation
(well, only when it is devirtualized).
Due to introduction of user-defined suffixes C++11 requires that there
is a space between literal and identifier to avoid ambiguity.
This patch makes the whole kernel build successfully with C++11.
This patch makes it possible to inline rdmsr and wrmsr instruction. The
performance impact shouldn't be significant since they are used relatively
rarely and wrmsr is usually a serializing instruction, but there is no reason
not to do so.
The goal of this patch is to amortize the cost of context switch by making
the compiler aware that context switch clobbers all registers. Because all
register need to be saved anyway there is no additional cost of using
callee saved register in the function that does the context switch.
Similarly to previous patch regarding GDT this is mostly a rewrite of
IDT handling code from C to C++. Thanks to constexpr IDT is now entirely
generated at compile-time.
Virtually no functional change, just rewriting the code from
"C in *.cpp files" to C++. Use of constexpr may be advantageous but
that code is not performance critical anyway.
While resolving TLS related relocations it is necessary to know the DSO
that defines the symbol. Without proper support in caching that information
is available only when the symbol is resolved first time. That works well
for TLS since TLS_DTPMOD is guaranteed to be before TLS_DTPOFF relocation.
This patch makes the newly introduced parts of the interface work in a
general case.
Previously TLS_DTPMOD relocation blindly returned ID of the current DSO.
This patch does proper symbol lookup if there is a symbol assigned to the
relocation and uses ID of the DSO in which the symbol is defined.
This patch introduces support of ELF based TLS handling with lazy allocation
and initalization of TLS block for each DSO and thread. The implementation
generally follows the official ABI except that generation counter in dtv
is in fact a pointer to Generation object that contains both generation
counter and size of the dtv. That simplified the implementation a bit, but
could be changed later. The ABI requirements regariding in memory position
of TLS block is not honoured what results in static TLS model being
unsupported. However, that should not be a problem as long as
"executables" in Haiku are in fact shared objects and optimizations which
require specific TLS block in memory layout are not possible anyway.
The scheduler expects that all threads expect the initial idle threads
have priority in range [THREAD_MIN_SET_PRIORITY, THREAD_MAX_SET_PRIORITY].
If the requested pririty is out of range the value is clamped. Failing
with B_BAD_VALUE is probably an overkill since there isn't any real
change in the guarantees provided by the scheduler about the behavior
of such thread. Also, BeBook suggests that spawn_thread() can specify
priority 0.
This patch remove the old thread migration logic which used few special
cases and (broken) general check that attempted to balance threads.
The new logic is pretty straightforward and seems perform well without
any additional special cases. Current core is compared with the least loaded
one and the thread is migrated if that would result in estimated loads of
both cores (i.e. the current one and the least loaded one) to become closer
to the average load (i.e. average of that two cores).
Currently, ThreadData::ShouldRebalance() (and mode specific functions
it calls) only decides whether to migrate thread to another core or not.
However, in most cases it actually needs to find the best candidate for
new core so it could as well return that information.
After load_image() the child thread is suspended and the parent is
expected to resume it later. However, it is possible that the parent
attempts to resume its child after it has been notified that the image
had been loaded but before the child managed to suspend itself. In such
case the child would suspends itself after that wake up attempt and,
consequently will not be ever resumed.
To mitigate that problem flag Thread::going_to_suspend has been added
which helps synchronizing thread suspension and continuation in a similar
way that "traditional" thread blocking is performed. This means that
the child should behave in a following manner: set its going_to_suspend flag,
notify the parent (i.e. any thread that may want to resume it), acquire
its scheduler_lock and suspend itself if the going_to_suspend flag is set.
The parent should follow pattern: clear going_to_suspend flag of the thread
that is about to be resumed, acquire that thread scheduler_lock and enqueue
it in a run queue if it is suspended.
Thanks Oliver for reporting the bug and identifying what causes it.
Most of the actual UserEvent work is done in DPC so that we don't have
to care about the limitations of the context in which UserEvent::Fire()
is invoked. This requires appropriate management of lifetime of UserEvent
instances to make sure that DoDPC() method is always called on a valid
object.
In low latency mode the scheduler would not attempt to balance load
on not heavily loaded cores unless difference in load exceeded
kLoadDifference * 2 (i.e. 40 percentage points), which does not seem
to be good enough.
To make sure that load statistics are accurate on idle cores each time
idle thread is scheduled a timer is set to update load when current
load measurement interval elapses. However, core load is defined as the
average load during last measurement interval and idle core may be still
considered busy if it was not idle during entire measurement interval.
Since, load update timer is a one shot timer that information will not be
updated until the core becomes active again.
To mitigate that issue load update timer is set to fire after two load
measurement intervals had elapsed.
Should fix#10628. If there is a race condition with a writer getting
minimum or maximum from double ended heap may incorrectly result NULL.
Which is not expected in the most of the thread migration logic. Apart
from that, because of the race condition heap state may be observed as
inconsistent thus failing assertions.
ended heap
The main purpose of using atomic_get() was the necessity of a compiler
barrier to prevent the compiler from optimizing busy loops. However,
each such loop contains in its body at least one statement that acts
as a compiler barrier (namely, cpu_wait() or cpu_pause()) making
atomic_get() redundant (well, atomic_get() is stronger - it also issues
a load barrier but in these particular cases we do not need it).
If the initial attempt to acquire read spinlock fails we use more relaxed
loop (which doesn't require CPU to lock the bus). However, check in that
loop, incorrectly, didn't allow a lock to be acquired when there was at
least one other reader.
NFS4 specification requires that modified data has to be flushed before
sending CLOSE request. Our original implementation was stricter than that
by flushing all modified data at each file close. That caused problems
with e.g. Tracker, which likes to periodically open files for read only,
in a situation when there were still asynchronous writes pending
(e.g. large file copied).
The initial core assignment has to be done without any knowledge about
the thread behaviour. Moreover, short lived tasks may spent most of their
time executing on that initial core. This patch attempts to improve the
qualiti of that initial decision.
The main purpose of this patch is to eliminate the delay between thread
migration and result of that migration being visible in load statistics.
Such delay, in certain circumstances, may cause some cores to become
overloaded because the scheduler migrates too many threads to them before
the effect of migration becomes apparent.
In order to keep the scheduler tickless core load is computed and updated
only during various scheduler events (i.e. thread enqueue, reschedule, etc).
The problem it creates is that if a core becomes idle its load may remain
outdated for an extended period of time thus resulting in suboptimal thread
migration decisions.
The solution to this problem is to add a timer each time an idle thread is
scheudled which, after kLoadMeasureInterval, would fire and force load
update.
Priority penalties were made more strict in order to prevent situation
when two or more high priority threads uses up all available CPU time
in such manner that they do not receive a penalty but starve low priority
threads.
However, a significant change to thread priorites has been made since and
now priority of all non real time threads varies in a range from 1 to
static priority minus penalty. This means that the scheduler is able to
prevent thread starvation without any complex penalty policies.
Originially, core load was a sum of eastimated loads of all currently
running or ready threads on a given core. Such value is changing very
rapidly preventing the thread migration logic from making any reasonable
decisions.
This patch changes the way core load is computed to make it more stable
thus improving the qualitiy of decisions made by the thread migration logic.
Currently core load is a sum of estimated loads of all threads that have been
ready during last load measurement interval and haven't been migrated or
killed.
The main reason for this patch is to fix gcc 4.8.2 warning about
hierarchyLevels possibly being used not initialized. Such thing
actually can not happen since all x2APIC CPUs are aware of at least
3 topology levels. However, once more topology levels are introduced
we will have to deal with CPUs that do not report information about all
of them.
User timers may cause another thread to become ready in which case we would
like this to happen before scheduler_reschedule() chooses next thread to
be executed.
UserEvent can be fired from scheduler_reschedule() i.e. while holding current
thread scheduler_lock. If the current thread goes sleep and during reschedule
one of its timers sends a signel to it, then scheduler_enqueue_in_run_queue()
attempts to acquire again its scheduler_lock resulting in a deadlock.
There was also a minor issue with both scheduler_reschedule() and
scheduler_enqueue_in_run_queue() acquiring current CPU scheduler mode lock.
On multisocket systems as well as under virtual machines logical CPUs
may use separate TSC. We could attempt to synchronize them what probably
would solve problems on multisocket systems. Unfortunately, when running
under hypervisor there is still a chance that TSC will get out of sync
again (e.g. cpufreq enabled on host when there is no invariant TSC). As
long as we use RDTSC as our main time source the scheduler must accept the
fact that time may go backwards (what isn't really a serious problem).
Paweł Dziepak